Jul 15 16:16:36.556598 ip-10-0-136-51 systemd[1]: Starting Kubernetes Kubelet... Jul 15 16:16:38.043653 ip-10-0-136-51 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 16:16:38.043653 ip-10-0-136-51 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 16:16:38.043653 ip-10-0-136-51 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 16:16:38.043653 ip-10-0-136-51 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 16:16:38.043653 ip-10-0-136-51 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 16:16:38.045834 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.045752 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 16:16:38.053127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053113 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 16:16:38.053127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053127 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053131 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053135 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053138 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053141 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053144 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053147 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053150 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053153 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053155 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053158 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053160 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053164 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053166 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053169 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053172 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053174 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053177 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053179 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 16:16:38.053189 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053182 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053185 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053187 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053190 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053192 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053195 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053198 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053201 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053203 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053206 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053208 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053211 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053213 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053216 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053219 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053221 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053224 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053226 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053229 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053232 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 16:16:38.053629 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053234 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053237 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053239 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053241 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053244 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053246 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053249 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053251 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053254 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053256 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053258 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053261 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053263 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053266 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053269 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053272 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053275 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053277 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053280 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053282 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 16:16:38.054116 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053286 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053290 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053293 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053296 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053299 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053301 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053304 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053306 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053309 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053313 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053316 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053320 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053323 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053326 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053330 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053333 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053336 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053339 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053342 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 16:16:38.054597 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053345 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053347 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053350 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053354 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053356 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053359 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053362 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053365 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053368 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053371 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053374 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053376 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053379 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053382 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053384 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053387 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053390 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053392 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053395 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.053398 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 16:16:38.055063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054092 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054098 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054101 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054105 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054107 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054110 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054113 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054116 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054120 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054122 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054125 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054128 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054131 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054133 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054136 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054139 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054141 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054144 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054146 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 16:16:38.055532 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054150 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054168 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054172 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054176 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054179 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054182 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054185 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054188 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054191 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054194 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054196 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054199 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054202 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054204 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054207 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054209 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054212 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054214 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054216 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054219 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 16:16:38.056012 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054222 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054225 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054228 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054233 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054235 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054238 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054241 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054243 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054245 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054248 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054250 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054253 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054256 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054260 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054263 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054265 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054268 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054270 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054273 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 16:16:38.056494 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054275 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054278 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054280 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054283 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054285 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054288 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054290 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054293 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054296 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054298 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054301 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054303 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054305 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054309 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054313 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054315 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054318 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054320 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054323 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 16:16:38.056966 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054325 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054329 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054332 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054335 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054338 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054340 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054344 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054346 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054349 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054352 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054354 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054357 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054359 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054361 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054364 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054367 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054369 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054371 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054374 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 16:16:38.057422 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054376 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054380 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054383 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054793 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054800 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054803 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054807 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054810 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054813 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054816 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054818 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054821 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054824 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054826 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054829 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054831 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054834 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054836 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054839 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 16:16:38.057887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054842 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054845 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054848 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054851 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054854 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054856 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054859 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054862 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054865 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054882 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054884 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054887 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054889 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054892 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054895 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054898 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054900 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054902 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054905 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054907 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 16:16:38.058395 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054910 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054913 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054916 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054918 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054920 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054923 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054926 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054928 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054931 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054933 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054936 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054939 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054942 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054945 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054948 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054951 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054953 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054956 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054958 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054961 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 16:16:38.058882 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054964 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054966 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054969 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054971 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054973 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054980 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054982 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054985 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054988 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054990 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054992 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054995 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.054997 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055000 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055003 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055006 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055008 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055011 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055013 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055015 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 16:16:38.059364 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055018 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055021 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055023 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055025 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055028 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055031 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055033 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055035 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055038 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055040 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055043 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055046 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055049 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055051 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055055 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055059 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055062 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055065 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055068 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 16:16:38.059931 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055070 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055073 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055076 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055080 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055149 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055158 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055166 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055172 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055180 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055185 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055190 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055194 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055197 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055200 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055204 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055207 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055210 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055213 2560 flags.go:64] FLAG: --cgroup-root="" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055216 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055219 2560 flags.go:64] FLAG: --client-ca-file="" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055222 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055225 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055229 2560 flags.go:64] FLAG: --cluster-domain="" Jul 15 16:16:38.060385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055232 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055235 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055238 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055241 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055244 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055247 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055250 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055253 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055256 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055259 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055262 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055265 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055268 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055272 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055275 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055278 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055281 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055284 2560 flags.go:64] FLAG: --enable-server="true" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055296 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055303 2560 flags.go:64] FLAG: --event-burst="100" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055306 2560 flags.go:64] FLAG: --event-qps="50" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055308 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055311 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055314 2560 flags.go:64] FLAG: --eviction-hard="" Jul 15 16:16:38.061375 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055318 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055321 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055326 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055329 2560 flags.go:64] FLAG: --eviction-soft="" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055332 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055334 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055337 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055340 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055343 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055346 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055348 2560 flags.go:64] FLAG: --feature-gates="" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055352 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055355 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055358 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055360 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055363 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055366 2560 flags.go:64] FLAG: --help="false" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055369 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055372 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055375 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055378 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055381 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055385 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055387 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 16:16:38.062232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055390 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055393 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055395 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055399 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055402 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055405 2560 flags.go:64] FLAG: --kube-reserved="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055408 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055410 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055413 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055416 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055420 2560 flags.go:64] FLAG: --lock-file="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055423 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055426 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055429 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055434 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055437 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055440 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055443 2560 flags.go:64] FLAG: --logging-format="text" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055446 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055449 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055451 2560 flags.go:64] FLAG: --manifest-url="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055454 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055459 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055461 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055465 2560 flags.go:64] FLAG: --max-pods="110" Jul 15 16:16:38.063280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055468 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055471 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055474 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055477 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055479 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055482 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055485 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055492 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055495 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055497 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055500 2560 flags.go:64] FLAG: --pod-cidr="" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055504 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055509 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055512 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055515 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055517 2560 flags.go:64] FLAG: --port="10250" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055520 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055523 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f69ba4817ca94434" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055527 2560 flags.go:64] FLAG: --qos-reserved="" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055530 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055533 2560 flags.go:64] FLAG: --register-node="true" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055536 2560 flags.go:64] FLAG: --register-with-taints="" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055540 2560 flags.go:64] FLAG: --registry-burst="10" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055543 2560 flags.go:64] FLAG: --registry-qps="5" Jul 15 16:16:38.064384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055546 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055548 2560 flags.go:64] FLAG: --reserved-memory="" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055552 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055555 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055558 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055561 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055563 2560 flags.go:64] FLAG: --runonce="false" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055566 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055569 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055572 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055575 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055577 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055581 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055585 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055588 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055590 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055593 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055596 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055598 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055601 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055610 2560 flags.go:64] FLAG: --system-cgroups="" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055613 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055618 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055621 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055623 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 16:16:38.065394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055630 2560 flags.go:64] FLAG: --tls-min-version="" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055633 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055636 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055639 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055642 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055645 2560 flags.go:64] FLAG: --v="2" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055648 2560 flags.go:64] FLAG: --version="false" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055652 2560 flags.go:64] FLAG: --vmodule="" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055655 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.055658 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055789 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055793 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055804 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055807 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055811 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055814 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055817 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055820 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055822 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055825 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055832 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055835 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 16:16:38.066188 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055837 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055840 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055842 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055845 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055847 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055850 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055853 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055857 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055859 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055862 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055864 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055881 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055883 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055886 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055889 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055891 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055894 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055896 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055899 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055901 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 16:16:38.067169 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055904 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055908 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055912 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055915 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055918 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055921 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055924 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055927 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055930 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055933 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055938 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055941 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055944 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055947 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055950 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055953 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055955 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055958 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055961 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 16:16:38.067691 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055964 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055967 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055969 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055972 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055976 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055979 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055981 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055984 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055987 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055989 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055992 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055995 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055997 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.055999 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056002 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056004 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056007 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056009 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056012 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056015 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 16:16:38.068284 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056018 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056021 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056023 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056027 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056029 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056032 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056035 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056038 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056040 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056043 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056045 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056048 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056050 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056053 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056056 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056058 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056062 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056065 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056068 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 16:16:38.068797 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056071 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056073 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056076 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056079 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056081 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056084 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056088 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056091 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.056094 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.056773 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.063781 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.063808 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.063834 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.063843 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 16:16:38.069337 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063946 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063955 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063961 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063966 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063971 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063976 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063980 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063988 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063993 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.063998 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064003 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064008 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064012 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064017 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064021 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064026 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064030 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064034 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064038 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 16:16:38.069689 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064043 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064047 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064051 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064055 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064059 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064063 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064067 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064072 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064076 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064080 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064086 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064090 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064097 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064104 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064108 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064113 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064117 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064121 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064125 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 16:16:38.070282 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064129 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064134 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064138 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064142 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064146 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064150 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064154 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064158 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064163 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064167 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064170 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064174 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064179 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064183 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064187 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064191 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064195 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064200 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064204 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064208 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064212 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 16:16:38.070777 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064216 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064220 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064226 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064236 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064240 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064244 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064248 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064252 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064256 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064260 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064264 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064268 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064272 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064277 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064281 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064285 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064290 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064294 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064298 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064302 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 16:16:38.071320 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064306 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064310 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064315 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064319 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064323 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064327 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064331 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064335 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064342 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064347 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064352 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064357 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064361 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064365 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064370 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064375 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064379 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064383 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064387 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 16:16:38.072127 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064390 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.064398 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064571 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064579 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064583 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064587 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064594 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064599 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064604 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064608 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064612 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064616 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064620 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064624 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064628 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 16:16:38.072887 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064632 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064636 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064640 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064644 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064649 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064654 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064658 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064662 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064666 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064670 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064674 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064679 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064683 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064689 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064694 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064699 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064704 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064708 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064713 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 16:16:38.073489 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064717 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064721 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064725 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064729 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064732 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064736 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064740 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064744 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064749 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064753 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064757 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064761 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064765 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064769 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064773 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064777 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064781 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064785 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064789 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064793 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 16:16:38.073991 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064798 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064801 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064805 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064810 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064814 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064818 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064823 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064826 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064830 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064836 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064840 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064844 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064848 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064852 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064856 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064860 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064864 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064884 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064888 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064892 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 16:16:38.074556 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064895 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064899 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064903 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064906 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064910 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064914 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064918 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064921 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064925 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064929 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064933 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064936 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064940 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064944 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064948 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064952 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064956 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064961 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064966 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064970 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 16:16:38.075063 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064974 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064978 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064983 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064987 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064991 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.064998 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:38.065003 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.065011 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.065148 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.067709 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.069092 2560 server.go:1019] "Starting client certificate rotation" Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.069198 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.069231 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 16:16:38.075568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.071995 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 16:16:38.075940 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.075445 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 16:16:38.087752 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.087733 2560 log.go:25] "Validated CRI v1 runtime API" Jul 15 16:16:38.089716 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.089700 2560 log.go:25] "Validated CRI v1 image API" Jul 15 16:16:38.092498 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.092483 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 16:16:38.095054 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.095035 2560 fs.go:135] Filesystem UUIDs: map[693b0c36-023e-41ff-b1a3-6d06812f211e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c3174f87-877e-4522-b4da-3179297a3050:/dev/nvme0n1p4] Jul 15 16:16:38.095114 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.095053 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 16:16:38.100482 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.100386 2560 manager.go:216] Machine: {Timestamp:2026-07-15 16:16:38.09914956 +0000 UTC m=+0.354049763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108330 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee37f99c445697c4a854457fbb74d SystemUUID:ec2ee37f-99c4-4569-7c4a-854457fbb74d BootID:32c6e7f4-e318-4d79-9e0e-6330a850f6c0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:54:b0:50:f2:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:54:b0:50:f2:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:91:7b:84:d7:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 16:16:38.100482 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.100481 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 16:16:38.100592 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.100556 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 16:16:38.104073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.104048 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 16:16:38.104202 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.104075 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 16:16:38.104858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.104848 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 16:16:38.104899 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.104860 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 16:16:38.104899 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.104878 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 16:16:38.105648 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.105639 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 16:16:38.107041 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.107027 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 16:16:38.109997 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.109986 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 15 16:16:38.110111 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110102 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 16:16:38.110145 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110132 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 15 16:16:38.110145 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110144 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 16:16:38.110199 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110168 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 16:16:38.110199 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110181 2560 kubelet.go:398] "Adding apiserver pod source" Jul 15 16:16:38.110199 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110191 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 16:16:38.110578 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110565 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 16:16:38.110620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.110592 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 16:16:38.113108 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.113083 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 16:16:38.115489 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.115055 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 16:16:38.115606 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.115587 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 16:16:38.116012 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.115998 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 16:16:38.116088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116016 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 16:16:38.116088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116032 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 16:16:38.116088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116038 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 16:16:38.116088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116047 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 16:16:38.116088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116053 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 16:16:38.116325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116289 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 16:16:38.116697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116538 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 16:16:38.116697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116558 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 16:16:38.116697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116590 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 16:16:38.116697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116602 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 16:16:38.116697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116618 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 16:16:38.116697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.116638 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 16:16:38.118166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.118145 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 16:16:38.118166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.118168 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 16:16:38.121542 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.121524 2560 server.go:1268] "Started kubelet" Jul 15 16:16:38.121616 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.121584 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 16:16:38.121685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.121596 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 16:16:38.121726 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.121705 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 16:16:38.122423 ip-10-0-136-51 systemd[1]: Started Kubernetes Kubelet. Jul 15 16:16:38.123104 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.122834 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 16:16:38.123914 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.123899 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 15 16:16:38.127153 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.127134 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 16:16:38.127228 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.127162 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 16:16:38.127228 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.127159 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 16:16:38.128345 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.128322 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 16:16:38.130495 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.130478 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 16:16:38.130495 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.130488 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 16:16:38.130618 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.130537 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.131241 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131221 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 16:16:38.131241 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131223 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 16:16:38.131392 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131253 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 16:16:38.131392 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131316 2560 factory.go:55] Registering systemd factory Jul 15 16:16:38.131392 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131331 2560 factory.go:223] Registration of the systemd container factory successfully Jul 15 16:16:38.131392 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131334 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 15 16:16:38.131392 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131342 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 15 16:16:38.131605 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131543 2560 factory.go:153] Registering CRI-O factory Jul 15 16:16:38.131605 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131556 2560 factory.go:223] Registration of the crio container factory successfully Jul 15 16:16:38.131697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131612 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 16:16:38.131697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131640 2560 factory.go:103] Registering Raw factory Jul 15 16:16:38.131697 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.131656 2560 manager.go:1196] Started watching for new ooms in manager Jul 15 16:16:38.132356 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.132336 2560 manager.go:349] Starting recovery of all containers Jul 15 16:16:38.134050 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.134023 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 16:16:38.134176 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.134159 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 16:16:38.135139 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.134024 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-51.ec2.internal.18c282c80d5e5f3f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-51.ec2.internal,UID:ip-10-0-136-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-51.ec2.internal,},FirstTimestamp:2026-07-15 16:16:38.121496383 +0000 UTC m=+0.376396582,LastTimestamp:2026-07-15 16:16:38.121496383 +0000 UTC m=+0.376396582,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-51.ec2.internal,}" Jul 15 16:16:38.144864 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.144714 2560 manager.go:354] Recovery completed Jul 15 16:16:38.149770 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.149757 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.151904 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.151881 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.151990 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.151922 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.151990 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.151933 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.152376 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.152359 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 16:16:38.152376 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.152373 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.152513 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.152374 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 16:16:38.152513 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.152449 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 15 16:16:38.153041 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.153021 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.153097 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.153057 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.153097 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.153073 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.153601 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.153590 2560 policy_none.go:49] "None policy: Start" Jul 15 16:16:38.153635 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.153607 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 16:16:38.153635 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.153617 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 16:16:38.154521 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.154457 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-51.ec2.internal.18c282c80f2e5486 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-51.ec2.internal,UID:ip-10-0-136-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-51.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-51.ec2.internal,},FirstTimestamp:2026-07-15 16:16:38.151902342 +0000 UTC m=+0.406802541,LastTimestamp:2026-07-15 16:16:38.151902342 +0000 UTC m=+0.406802541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-51.ec2.internal,}" Jul 15 16:16:38.154881 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.154857 2560 policy_none.go:47] "Start" Jul 15 16:16:38.165625 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.165568 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-51.ec2.internal.18c282c80f2eb5d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-51.ec2.internal,UID:ip-10-0-136-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-51.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-51.ec2.internal,},FirstTimestamp:2026-07-15 16:16:38.151927252 +0000 UTC m=+0.406827448,LastTimestamp:2026-07-15 16:16:38.151927252 +0000 UTC m=+0.406827448,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-51.ec2.internal,}" Jul 15 16:16:38.168939 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.168923 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8lr2v" Jul 15 16:16:38.178789 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.178768 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8lr2v" Jul 15 16:16:38.183743 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.183691 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-51.ec2.internal.18c282c80f2ed9f7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-51.ec2.internal,UID:ip-10-0-136-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-51.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-51.ec2.internal,},FirstTimestamp:2026-07-15 16:16:38.151936503 +0000 UTC m=+0.406836700,LastTimestamp:2026-07-15 16:16:38.151936503 +0000 UTC m=+0.406836700,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-51.ec2.internal,}" Jul 15 16:16:38.192407 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192391 2560 manager.go:337] "Starting Device Plugin manager" Jul 15 16:16:38.192470 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.192418 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 16:16:38.192470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192426 2560 server.go:87] "Starting device plugin registration server" Jul 15 16:16:38.192616 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192603 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 16:16:38.192653 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192620 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 16:16:38.192712 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192702 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 16:16:38.192777 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192765 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 16:16:38.192827 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.192779 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 16:16:38.193309 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.193292 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 16:16:38.193407 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.193338 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.293370 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.293348 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.294466 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.294137 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.294466 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.294167 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.294466 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.294180 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.294466 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.294206 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.302514 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.302490 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 16:16:38.303710 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.303690 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 16:16:38.303815 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.303725 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 16:16:38.303815 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.303746 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 16:16:38.303815 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.303776 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 16:16:38.309030 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.308977 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 16:16:38.318153 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.318140 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.318212 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.318160 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-51.ec2.internal\": node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.404128 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.404110 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal"] Jul 15 16:16:38.404247 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.404175 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.405572 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.405557 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.405634 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.405591 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.405634 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.405604 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.406769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.406757 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.406914 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.406899 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.406981 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.406938 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.407430 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.407410 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.407517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.407434 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.407517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.407444 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.407517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.407470 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.407517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.407489 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.407517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.407497 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.408681 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.408668 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.408735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.408691 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:38.409245 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.409230 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:38.409321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.409251 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:38.409321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.409264 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:38.423600 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.423583 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.433754 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.433738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e5d0398c51c2ed143c889c77c2dfab2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal\" (UID: \"5e5d0398c51c2ed143c889c77c2dfab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.433831 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.433763 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e5d0398c51c2ed143c889c77c2dfab2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal\" (UID: \"5e5d0398c51c2ed143c889c77c2dfab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.433831 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.433780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0220f44d9b57d66146d6849b5a148314-config\") pod \"kube-apiserver-proxy-ip-10-0-136-51.ec2.internal\" (UID: \"0220f44d9b57d66146d6849b5a148314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.439444 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.439428 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-51.ec2.internal\" not found" node="ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.443816 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.443786 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-51.ec2.internal\" not found" node="ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.524070 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.524045 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.534460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.534440 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e5d0398c51c2ed143c889c77c2dfab2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal\" (UID: \"5e5d0398c51c2ed143c889c77c2dfab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.534517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.534466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e5d0398c51c2ed143c889c77c2dfab2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal\" (UID: \"5e5d0398c51c2ed143c889c77c2dfab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.534517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.534485 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0220f44d9b57d66146d6849b5a148314-config\") pod \"kube-apiserver-proxy-ip-10-0-136-51.ec2.internal\" (UID: \"0220f44d9b57d66146d6849b5a148314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.534579 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.534525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e5d0398c51c2ed143c889c77c2dfab2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal\" (UID: \"5e5d0398c51c2ed143c889c77c2dfab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.534579 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.534544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0220f44d9b57d66146d6849b5a148314-config\") pod \"kube-apiserver-proxy-ip-10-0-136-51.ec2.internal\" (UID: \"0220f44d9b57d66146d6849b5a148314\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.534579 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.534525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e5d0398c51c2ed143c889c77c2dfab2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal\" (UID: \"5e5d0398c51c2ed143c889c77c2dfab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.625111 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.625064 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.726104 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.726080 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.741609 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.741593 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.747369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:38.747352 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" Jul 15 16:16:38.826695 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.826671 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:38.927269 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:38.927214 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:39.027710 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:39.027677 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:39.069216 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.069189 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 16:16:39.069758 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.069365 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 16:16:39.127857 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:39.127834 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:39.182144 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.182071 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 16:11:38 +0000 UTC" deadline="2028-04-17 15:54:16.641452342 +0000 UTC" Jul 15 16:16:39.182144 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.182103 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15407h37m37.459353036s" Jul 15 16:16:39.227924 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:39.227903 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:39.228276 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:39.228255 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5d0398c51c2ed143c889c77c2dfab2.slice/crio-d3ed8587410396d70d50dd4669f865da1fe5d36bbac50f8ef6fe675b19c46731 WatchSource:0}: Error finding container d3ed8587410396d70d50dd4669f865da1fe5d36bbac50f8ef6fe675b19c46731: Status 404 returned error can't find the container with id d3ed8587410396d70d50dd4669f865da1fe5d36bbac50f8ef6fe675b19c46731 Jul 15 16:16:39.232931 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.232919 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 16:16:39.306512 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.306474 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" event={"ID":"5e5d0398c51c2ed143c889c77c2dfab2","Type":"ContainerStarted","Data":"d3ed8587410396d70d50dd4669f865da1fe5d36bbac50f8ef6fe675b19c46731"} Jul 15 16:16:39.328610 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:39.328591 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-51.ec2.internal\" not found" Jul 15 16:16:39.363097 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.363082 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 16:16:39.364085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.364069 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientMemory" Jul 15 16:16:39.364168 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.364103 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 16:16:39.364168 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.364118 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeHasSufficientPID" Jul 15 16:16:39.370287 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.370269 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 16:16:39.431185 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.431138 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" Jul 15 16:16:39.463385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.463332 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 16:16:39.465608 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.465593 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" Jul 15 16:16:39.480948 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.480930 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 16:16:39.590061 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.590039 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 16:16:39.646747 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.646725 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 16:16:39.975035 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:39.975006 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 16:16:40.111270 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.111241 2560 apiserver.go:52] "Watching apiserver" Jul 15 16:16:40.119883 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.119845 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 16:16:40.120234 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.120209 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-65mm5","kube-system/global-pull-secret-syncer-ks4kq","kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal","openshift-dns/node-resolver-78lgd","openshift-image-registry/node-ca-495xf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal","openshift-multus/multus-additional-cni-plugins-vtfzn","openshift-multus/multus-pgpcx","kube-system/konnectivity-agent-rcbg5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p","openshift-cluster-node-tuning-operator/tuned-twdwz","openshift-multus/network-metrics-daemon-flp4t","openshift-network-diagnostics/network-check-target-cf9rh","openshift-network-operator/iptables-alerter-5ln4l"] Jul 15 16:16:40.121896 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.121859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.123143 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.123121 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:40.123253 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.123197 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:40.124380 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.124355 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.125586 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.125566 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.126932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.126912 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.128344 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.128285 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.128471 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.128454 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 16:16:40.128855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.128682 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 16:16:40.128855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.128683 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 16:16:40.128855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.128763 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.128855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.128763 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.129390 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.129372 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nngdk\"" Jul 15 16:16:40.129643 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.129623 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.130467 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.130445 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.130549 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.130520 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 16:16:40.131147 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.131067 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.131661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.131643 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 16:16:40.132625 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.132503 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.132625 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.132515 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 16:16:40.132625 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.132580 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:40.133405 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.133388 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.133527 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.133510 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.133587 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.133522 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.133908 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.133887 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bl22k\"" Jul 15 16:16:40.134045 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.134027 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.134174 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.134156 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5wphh\"" Jul 15 16:16:40.134311 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.134289 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ls8w2\"" Jul 15 16:16:40.134398 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.134299 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.135336 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.135237 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.137784 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.136714 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.138642 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.138622 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.138737 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.138688 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:40.140391 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.140367 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.140864 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.140575 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 16:16:40.140864 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.140690 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r2758\"" Jul 15 16:16:40.140864 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.140782 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nbbnk\"" Jul 15 16:16:40.141103 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141000 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 16:16:40.141103 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141023 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hcm2g\"" Jul 15 16:16:40.141336 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141320 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 16:16:40.141567 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141542 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 16:16:40.141567 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141561 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 16:16:40.141713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141549 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.141713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.141549 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 16:16:40.144324 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvlh\" (UniqueName: \"kubernetes.io/projected/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-kube-api-access-smvlh\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.144421 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144331 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-hostroot\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.144421 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144345 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9-agent-certs\") pod \"konnectivity-agent-rcbg5\" (UID: \"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9\") " pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.144421 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144364 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-cni-bin\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144421 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144415 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-cni-netd\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144453 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-node-log\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144498 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dxn\" (UniqueName: \"kubernetes.io/projected/ad2f5186-47e7-41a8-a6c3-91b8093440b3-kube-api-access-t5dxn\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144524 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9437532-8ae5-4ded-b3c5-987dffce18fa-hosts-file\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt7qf\" (UniqueName: \"kubernetes.io/projected/479fd180-dffc-45e3-8741-284cf398e817-kube-api-access-kt7qf\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-var-lib-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144640 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fv8\" (UniqueName: \"kubernetes.io/projected/f9437532-8ae5-4ded-b3c5-987dffce18fa-kube-api-access-j8fv8\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144679 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-conf-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144721 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-systemd\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144746 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-cnibin\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144803 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-socket-dir-parent\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144836 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-systemd-units\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144855 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-etc-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144884 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-system-cni-dir\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144908 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-host-slash\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144927 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-etc-kubernetes\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-run-netns\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.144987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.144975 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145000 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145033 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-env-overrides\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145097 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b77867d-4145-4515-945f-7a50589dd046-kubelet-config\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-slash\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145148 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cnibin\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-system-cni-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-os-release\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145206 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-host\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-cni-bin\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-multus-certs\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145307 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-cni-multus\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145324 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145341 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-os-release\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145363 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.145486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-iptables-alerter-script\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-cni-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145493 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b77867d-4145-4515-945f-7a50589dd046-dbus\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145520 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-k8s-cni-cncf-io\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145543 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-netns\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-kubelet\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/479fd180-dffc-45e3-8741-284cf398e817-multus-daemon-config\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145629 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-serviceca\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6njk\" (UniqueName: \"kubernetes.io/projected/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-kube-api-access-j6njk\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145671 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-ovn\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/479fd180-dffc-45e3-8741-284cf398e817-cni-binary-copy\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-kubelet\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145739 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-ovnkube-config\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145773 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5912f599-2ac6-4169-900f-e5576f6d6e67-ovn-node-metrics-cert\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145795 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-ovnkube-script-lib\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145817 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-log-socket\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.146122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9437532-8ae5-4ded-b3c5-987dffce18fa-tmp-dir\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.146700 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145860 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9-konnectivity-ca\") pod \"konnectivity-agent-rcbg5\" (UID: \"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9\") " pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.146700 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.145899 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792xz\" (UniqueName: \"kubernetes.io/projected/5912f599-2ac6-4169-900f-e5576f6d6e67-kube-api-access-792xz\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.153752 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.153730 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 16:16:40.153752 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.153748 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.153956 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.153773 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 16:16:40.153956 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.153843 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.154194 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.154168 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c5qhj\"" Jul 15 16:16:40.154254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.154216 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 16:16:40.154254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.154168 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-84c9g\"" Jul 15 16:16:40.154254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.154257 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 16:16:40.177149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.177128 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 16:16:40.232356 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.232282 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 16:16:40.247098 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.247220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247110 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-device-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.247220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247137 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-var-lib-kubelet\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.247220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247162 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-iptables-alerter-script\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.247220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247187 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-cni-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247254 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.247414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247309 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-systemd\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.247414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247335 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstz6\" (UniqueName: \"kubernetes.io/projected/8bed2434-1dfa-46d3-908f-b9a243db93a2-kube-api-access-wstz6\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.247414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247358 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b77867d-4145-4515-945f-7a50589dd046-dbus\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.247414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247366 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-cni-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247401 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-k8s-cni-cncf-io\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247429 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247437 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-netns\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247460 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-kubelet\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/479fd180-dffc-45e3-8741-284cf398e817-multus-daemon-config\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247551 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b77867d-4145-4515-945f-7a50589dd046-dbus\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-serviceca\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247592 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6njk\" (UniqueName: \"kubernetes.io/projected/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-kube-api-access-j6njk\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247608 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-ovn\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/479fd180-dffc-45e3-8741-284cf398e817-cni-binary-copy\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-kubelet\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247661 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-ovnkube-config\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247675 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5912f599-2ac6-4169-900f-e5576f6d6e67-ovn-node-metrics-cert\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.247685 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-ovnkube-script-lib\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-log-socket\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247730 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6v2l\" (UniqueName: \"kubernetes.io/projected/7220e675-998f-4d9a-8e39-e508eb158616-kube-api-access-v6v2l\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247748 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysctl-d\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247764 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9437532-8ae5-4ded-b3c5-987dffce18fa-tmp-dir\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247769 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-iptables-alerter-script\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247786 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9-konnectivity-ca\") pod \"konnectivity-agent-rcbg5\" (UID: \"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9\") " pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247769 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247811 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792xz\" (UniqueName: \"kubernetes.io/projected/5912f599-2ac6-4169-900f-e5576f6d6e67-kube-api-access-792xz\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-sys-fs\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247853 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-ovn\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247918 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-log-socket\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-netns\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248036 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-serviceca\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248077 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-k8s-cni-cncf-io\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248101 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-kubelet\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.247857 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-sys\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.248252 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248199 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9437532-8ae5-4ded-b3c5-987dffce18fa-tmp-dir\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-host\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248295 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-tmp\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248327 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-ovnkube-config\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248351 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9w6\" (UniqueName: \"kubernetes.io/projected/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-kube-api-access-dc9w6\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248369 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvlh\" (UniqueName: \"kubernetes.io/projected/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-kube-api-access-smvlh\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248389 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-hostroot\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248392 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-kubelet\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248433 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9-agent-certs\") pod \"konnectivity-agent-rcbg5\" (UID: \"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9\") " pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248453 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-hostroot\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248461 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-cni-bin\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-cni-netd\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248763 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/479fd180-dffc-45e3-8741-284cf398e817-multus-daemon-config\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248765 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-cni-netd\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248792 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248812 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248823 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/479fd180-dffc-45e3-8741-284cf398e817-cni-binary-copy\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248855 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9-konnectivity-ca\") pod \"konnectivity-agent-rcbg5\" (UID: \"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9\") " pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.249003 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248850 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-cni-bin\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248857 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248914 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-node-log\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dxn\" (UniqueName: \"kubernetes.io/projected/ad2f5186-47e7-41a8-a6c3-91b8093440b3-kube-api-access-t5dxn\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9437532-8ae5-4ded-b3c5-987dffce18fa-hosts-file\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.248988 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-node-log\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt7qf\" (UniqueName: \"kubernetes.io/projected/479fd180-dffc-45e3-8741-284cf398e817-kube-api-access-kt7qf\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-var-lib-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249063 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9437532-8ae5-4ded-b3c5-987dffce18fa-hosts-file\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249069 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysconfig\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249095 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-kubernetes\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249125 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fv8\" (UniqueName: \"kubernetes.io/projected/f9437532-8ae5-4ded-b3c5-987dffce18fa-kube-api-access-j8fv8\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-conf-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249186 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-systemd\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-var-lib-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.249823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-cnibin\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249272 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-socket-dir-parent\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249298 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-conf-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-systemd-units\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249351 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-systemd-units\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-etc-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249395 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-run-systemd\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249406 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-registration-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249438 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249458 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-modprobe-d\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249486 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-system-cni-dir\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-cnibin\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249515 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-host-slash\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-ovnkube-script-lib\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249556 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-multus-socket-dir-parent\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-host-slash\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249578 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-etc-openvswitch\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.250630 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-etc-kubernetes\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249599 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-system-cni-dir\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249628 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-run-netns\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249635 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-etc-kubernetes\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249685 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249689 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-run-netns\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249724 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249794 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-env-overrides\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249838 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249844 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249882 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249902 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b77867d-4145-4515-945f-7a50589dd046-kubelet-config\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249918 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-slash\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cnibin\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-socket-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.249986 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-run\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.251433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250041 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-lib-modules\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250066 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-system-cni-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250087 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-os-release\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250113 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5912f599-2ac6-4169-900f-e5576f6d6e67-host-slash\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-host\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250146 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-host\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250159 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysctl-conf\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cnibin\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-tuned\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250244 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-cni-bin\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250275 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-system-cni-dir\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250294 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-multus-certs\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250302 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250317 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-etc-selinux\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250327 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-os-release\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-cni-multus\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250415 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.252193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250460 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-os-release\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad2f5186-47e7-41a8-a6c3-91b8093440b3-os-release\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250645 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-cni-multus\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250684 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5912f599-2ac6-4169-900f-e5576f6d6e67-env-overrides\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-var-lib-cni-bin\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250717 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/479fd180-dffc-45e3-8741-284cf398e817-host-run-multus-certs\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.250732 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b77867d-4145-4515-945f-7a50589dd046-kubelet-config\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.250818 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.250915 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:40.750883859 +0000 UTC m=+3.005784063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.251457 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad2f5186-47e7-41a8-a6c3-91b8093440b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.252084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5912f599-2ac6-4169-900f-e5576f6d6e67-ovn-node-metrics-cert\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.252884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.252770 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9-agent-certs\") pod \"konnectivity-agent-rcbg5\" (UID: \"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9\") " pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.262294 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.262273 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:16:40.262294 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.262294 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:16:40.262465 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.262303 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:40.262465 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.262360 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:40.762337203 +0000 UTC m=+3.017237388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:40.264447 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.264426 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvlh\" (UniqueName: \"kubernetes.io/projected/c9ffdfba-ae60-445a-96dd-30d45e9bf0f2-kube-api-access-smvlh\") pod \"iptables-alerter-5ln4l\" (UID: \"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2\") " pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.282021 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.281996 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt7qf\" (UniqueName: \"kubernetes.io/projected/479fd180-dffc-45e3-8741-284cf398e817-kube-api-access-kt7qf\") pod \"multus-pgpcx\" (UID: \"479fd180-dffc-45e3-8741-284cf398e817\") " pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.283184 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.283164 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-792xz\" (UniqueName: \"kubernetes.io/projected/5912f599-2ac6-4169-900f-e5576f6d6e67-kube-api-access-792xz\") pod \"ovnkube-node-65mm5\" (UID: \"5912f599-2ac6-4169-900f-e5576f6d6e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.294553 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.294533 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dxn\" (UniqueName: \"kubernetes.io/projected/ad2f5186-47e7-41a8-a6c3-91b8093440b3-kube-api-access-t5dxn\") pod \"multus-additional-cni-plugins-vtfzn\" (UID: \"ad2f5186-47e7-41a8-a6c3-91b8093440b3\") " pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.296075 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.296054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6njk\" (UniqueName: \"kubernetes.io/projected/af3759db-0951-4fdd-8f5d-b5ec058cbb3a-kube-api-access-j6njk\") pod \"node-ca-495xf\" (UID: \"af3759db-0951-4fdd-8f5d-b5ec058cbb3a\") " pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.297364 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.297346 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fv8\" (UniqueName: \"kubernetes.io/projected/f9437532-8ae5-4ded-b3c5-987dffce18fa-kube-api-access-j8fv8\") pod \"node-resolver-78lgd\" (UID: \"f9437532-8ae5-4ded-b3c5-987dffce18fa\") " pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.312055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.312032 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tg8vz" Jul 15 16:16:40.338288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.338267 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tg8vz" Jul 15 16:16:40.350962 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.350936 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9w6\" (UniqueName: \"kubernetes.io/projected/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-kube-api-access-dc9w6\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351048 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.350978 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351048 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351007 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysconfig\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysconfig\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351065 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-kubelet-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-kubernetes\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351130 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-registration-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351154 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-modprobe-d\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351197 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351205 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-kubernetes\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351209 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-registration-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351222 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-socket-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351245 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-run\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-lib-modules\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351281 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-modprobe-d\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351292 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysctl-conf\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.351298 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351317 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-socket-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351345 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-tuned\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.351360 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:40.851342457 +0000 UTC m=+3.106242649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351364 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-run\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-etc-selinux\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-device-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-var-lib-kubelet\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351474 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysctl-conf\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351489 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-systemd\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351477 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-lib-modules\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstz6\" (UniqueName: \"kubernetes.io/projected/8bed2434-1dfa-46d3-908f-b9a243db93a2-kube-api-access-wstz6\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-device-dir\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.351554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351551 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-var-lib-kubelet\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6v2l\" (UniqueName: \"kubernetes.io/projected/7220e675-998f-4d9a-8e39-e508eb158616-kube-api-access-v6v2l\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351584 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysctl-d\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-sys-fs\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351663 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-sys-fs\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351615 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7220e675-998f-4d9a-8e39-e508eb158616-etc-selinux\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351677 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-sysctl-d\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351696 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-sys\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351717 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-systemd\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-host\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351749 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-sys\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-tmp\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.352385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.351799 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-host\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.353609 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.353587 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-etc-tuned\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.353853 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.353831 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-tmp\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.368849 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.368825 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstz6\" (UniqueName: \"kubernetes.io/projected/8bed2434-1dfa-46d3-908f-b9a243db93a2-kube-api-access-wstz6\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.370039 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.370016 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6v2l\" (UniqueName: \"kubernetes.io/projected/7220e675-998f-4d9a-8e39-e508eb158616-kube-api-access-v6v2l\") pod \"aws-ebs-csi-driver-node-njt2p\" (UID: \"7220e675-998f-4d9a-8e39-e508eb158616\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.370521 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.370502 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9w6\" (UniqueName: \"kubernetes.io/projected/e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0-kube-api-access-dc9w6\") pod \"tuned-twdwz\" (UID: \"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0\") " pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.434593 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.434567 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" Jul 15 16:16:40.444442 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.444420 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5ln4l" Jul 15 16:16:40.453038 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.453017 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-78lgd" Jul 15 16:16:40.458710 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.458690 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-495xf" Jul 15 16:16:40.466071 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.466053 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgpcx" Jul 15 16:16:40.471855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.471832 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:16:40.478741 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.478719 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:16:40.483574 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.483522 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" Jul 15 16:16:40.488477 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.488460 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-twdwz" Jul 15 16:16:40.530143 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.530120 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7220e675_998f_4d9a_8e39_e508eb158616.slice/crio-5775f5beb516d710c2d662500cb801cbbdfb754fd51d90cce725ab51c4edd5d9 WatchSource:0}: Error finding container 5775f5beb516d710c2d662500cb801cbbdfb754fd51d90cce725ab51c4edd5d9: Status 404 returned error can't find the container with id 5775f5beb516d710c2d662500cb801cbbdfb754fd51d90cce725ab51c4edd5d9 Jul 15 16:16:40.531198 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.531171 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1252ca3_d2d7_4e5c_8f94_e40a530bf5c0.slice/crio-8d81b3951060185f9f798c3f614cba2ede6537f7edffb6c9a9dfad160c0a367f WatchSource:0}: Error finding container 8d81b3951060185f9f798c3f614cba2ede6537f7edffb6c9a9dfad160c0a367f: Status 404 returned error can't find the container with id 8d81b3951060185f9f798c3f614cba2ede6537f7edffb6c9a9dfad160c0a367f Jul 15 16:16:40.535601 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.535580 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479fd180_dffc_45e3_8741_284cf398e817.slice/crio-f898cc84588ad0d0f531b45730f24ce4413f0605085da2a07028e361a572d1e0 WatchSource:0}: Error finding container f898cc84588ad0d0f531b45730f24ce4413f0605085da2a07028e361a572d1e0: Status 404 returned error can't find the container with id f898cc84588ad0d0f531b45730f24ce4413f0605085da2a07028e361a572d1e0 Jul 15 16:16:40.536407 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.536381 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ffdfba_ae60_445a_96dd_30d45e9bf0f2.slice/crio-6ee156b8738970f26d634758605e224d455932fe640b5e51b4101da2f4eeacdc WatchSource:0}: Error finding container 6ee156b8738970f26d634758605e224d455932fe640b5e51b4101da2f4eeacdc: Status 404 returned error can't find the container with id 6ee156b8738970f26d634758605e224d455932fe640b5e51b4101da2f4eeacdc Jul 15 16:16:40.537405 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.537381 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9437532_8ae5_4ded_b3c5_987dffce18fa.slice/crio-fd682e9af22336ded061fee80aa99a0c08efd34b91ebae1d298bdc8a4c94b8c7 WatchSource:0}: Error finding container fd682e9af22336ded061fee80aa99a0c08efd34b91ebae1d298bdc8a4c94b8c7: Status 404 returned error can't find the container with id fd682e9af22336ded061fee80aa99a0c08efd34b91ebae1d298bdc8a4c94b8c7 Jul 15 16:16:40.538086 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.538066 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3759db_0951_4fdd_8f5d_b5ec058cbb3a.slice/crio-3b21e84da0c2e1a5cc15b8da18579f42700f3ed1edbb382f4b404478ef3584f5 WatchSource:0}: Error finding container 3b21e84da0c2e1a5cc15b8da18579f42700f3ed1edbb382f4b404478ef3584f5: Status 404 returned error can't find the container with id 3b21e84da0c2e1a5cc15b8da18579f42700f3ed1edbb382f4b404478ef3584f5 Jul 15 16:16:40.538987 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.538952 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5912f599_2ac6_4169_900f_e5576f6d6e67.slice/crio-cb17a19983dbb7e915095b89240fb5de77f317ba873987445a4659744e48d362 WatchSource:0}: Error finding container cb17a19983dbb7e915095b89240fb5de77f317ba873987445a4659744e48d362: Status 404 returned error can't find the container with id cb17a19983dbb7e915095b89240fb5de77f317ba873987445a4659744e48d362 Jul 15 16:16:40.539854 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.539821 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccd6b87_90bf_4d5e_86d4_867ac7bd23f9.slice/crio-950fdcdaa37254dd070ad3f951506df16ea4d88a99010dbd662b8dd8867f895a WatchSource:0}: Error finding container 950fdcdaa37254dd070ad3f951506df16ea4d88a99010dbd662b8dd8867f895a: Status 404 returned error can't find the container with id 950fdcdaa37254dd070ad3f951506df16ea4d88a99010dbd662b8dd8867f895a Jul 15 16:16:40.541143 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.541087 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2f5186_47e7_41a8_a6c3_91b8093440b3.slice/crio-75ebf2da3b63c6ff76aff50e94e249a4a0b2c6626bfe8d3c7c8f8c0a6751d7eb WatchSource:0}: Error finding container 75ebf2da3b63c6ff76aff50e94e249a4a0b2c6626bfe8d3c7c8f8c0a6751d7eb: Status 404 returned error can't find the container with id 75ebf2da3b63c6ff76aff50e94e249a4a0b2c6626bfe8d3c7c8f8c0a6751d7eb Jul 15 16:16:40.547718 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:16:40.547695 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0220f44d9b57d66146d6849b5a148314.slice/crio-175dd7e7c0ed14fd92a7200ca5d749da7a0542a30a09b394b95d530a3520c91f WatchSource:0}: Error finding container 175dd7e7c0ed14fd92a7200ca5d749da7a0542a30a09b394b95d530a3520c91f: Status 404 returned error can't find the container with id 175dd7e7c0ed14fd92a7200ca5d749da7a0542a30a09b394b95d530a3520c91f Jul 15 16:16:40.755056 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.754927 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:40.755135 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.755074 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:40.755179 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.755137 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:41.755119685 +0000 UTC m=+4.010019874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:40.855264 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.855237 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:40.855365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:40.855268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:40.855419 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.855372 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:40.855419 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.855404 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:16:40.855515 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.855425 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:16:40.855515 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.855438 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:40.855515 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.855414 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:41.855401168 +0000 UTC m=+4.110301351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:40.855515 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:40.855495 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:41.855479682 +0000 UTC m=+4.110379870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:41.305007 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.304574 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:41.305007 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.304700 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:41.326491 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.326418 2560 generic.go:356] "Generic (PLEG): container finished" podID="5e5d0398c51c2ed143c889c77c2dfab2" containerID="f7fa6aca36544b2c32448aeb9ae508fc3e15b615822b44b2df725e1a43ca7fad" exitCode=0 Jul 15 16:16:41.326624 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.326502 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" event={"ID":"5e5d0398c51c2ed143c889c77c2dfab2","Type":"ContainerDied","Data":"f7fa6aca36544b2c32448aeb9ae508fc3e15b615822b44b2df725e1a43ca7fad"} Jul 15 16:16:41.339757 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.339723 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" event={"ID":"0220f44d9b57d66146d6849b5a148314","Type":"ContainerStarted","Data":"175dd7e7c0ed14fd92a7200ca5d749da7a0542a30a09b394b95d530a3520c91f"} Jul 15 16:16:41.341014 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.340979 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 16:11:40 +0000 UTC" deadline="2027-12-11 13:42:51.737409561 +0000 UTC" Jul 15 16:16:41.341014 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.341005 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12333h26m10.396408596s" Jul 15 16:16:41.344917 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.344826 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerStarted","Data":"75ebf2da3b63c6ff76aff50e94e249a4a0b2c6626bfe8d3c7c8f8c0a6751d7eb"} Jul 15 16:16:41.365709 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.365682 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-495xf" event={"ID":"af3759db-0951-4fdd-8f5d-b5ec058cbb3a","Type":"ContainerStarted","Data":"3b21e84da0c2e1a5cc15b8da18579f42700f3ed1edbb382f4b404478ef3584f5"} Jul 15 16:16:41.371476 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.371447 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"cb17a19983dbb7e915095b89240fb5de77f317ba873987445a4659744e48d362"} Jul 15 16:16:41.379002 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.378978 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-78lgd" event={"ID":"f9437532-8ae5-4ded-b3c5-987dffce18fa","Type":"ContainerStarted","Data":"fd682e9af22336ded061fee80aa99a0c08efd34b91ebae1d298bdc8a4c94b8c7"} Jul 15 16:16:41.382176 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.382150 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcbg5" event={"ID":"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9","Type":"ContainerStarted","Data":"950fdcdaa37254dd070ad3f951506df16ea4d88a99010dbd662b8dd8867f895a"} Jul 15 16:16:41.384323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.384300 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5ln4l" event={"ID":"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2","Type":"ContainerStarted","Data":"6ee156b8738970f26d634758605e224d455932fe640b5e51b4101da2f4eeacdc"} Jul 15 16:16:41.398084 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.398036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgpcx" event={"ID":"479fd180-dffc-45e3-8741-284cf398e817","Type":"ContainerStarted","Data":"f898cc84588ad0d0f531b45730f24ce4413f0605085da2a07028e361a572d1e0"} Jul 15 16:16:41.405170 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.405143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-twdwz" event={"ID":"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0","Type":"ContainerStarted","Data":"8d81b3951060185f9f798c3f614cba2ede6537f7edffb6c9a9dfad160c0a367f"} Jul 15 16:16:41.420385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.420359 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" event={"ID":"7220e675-998f-4d9a-8e39-e508eb158616","Type":"ContainerStarted","Data":"5775f5beb516d710c2d662500cb801cbbdfb754fd51d90cce725ab51c4edd5d9"} Jul 15 16:16:41.762111 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.762068 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:41.762292 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.762230 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:41.762292 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.762290 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:43.762272527 +0000 UTC m=+6.017172714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:41.863124 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.863041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:41.863124 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:41.863089 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:41.863337 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.863226 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:41.863337 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.863290 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:43.863271645 +0000 UTC m=+6.118171852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:41.863716 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.863691 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:16:41.863716 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.863714 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:16:41.863852 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.863726 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:41.863852 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:41.863770 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:43.863754966 +0000 UTC m=+6.118655155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:42.304323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:42.304241 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:42.304469 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:42.304379 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:42.304787 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:42.304767 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:42.304918 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:42.304882 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:42.341943 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:42.341864 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 16:11:40 +0000 UTC" deadline="2028-01-25 06:59:22.827717783 +0000 UTC" Jul 15 16:16:42.341943 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:42.341920 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13406h42m40.485803639s" Jul 15 16:16:42.454412 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:42.454377 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" event={"ID":"5e5d0398c51c2ed143c889c77c2dfab2","Type":"ContainerStarted","Data":"3b0cca8cb0904075c3e07ba32f93d5ad08767f4d9144bc78159c2bc2e1ab5fe0"} Jul 15 16:16:43.304955 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:43.304448 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:43.304955 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.304593 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:43.778073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:43.777928 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:43.778073 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.778003 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:43.778549 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.778080 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:47.77806024 +0000 UTC m=+10.032960484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:43.878956 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:43.878918 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:43.879124 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:43.878972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:43.879192 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.879130 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:43.879267 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.879198 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:47.87917268 +0000 UTC m=+10.134072869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:43.879608 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.879586 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:16:43.879695 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.879613 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:16:43.879695 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.879626 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:43.879695 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:43.879676 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:47.879661227 +0000 UTC m=+10.134561425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:44.308338 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:44.308303 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:44.308497 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:44.308351 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:44.308569 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:44.308493 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:44.308913 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:44.308884 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:45.304029 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:45.303983 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:45.304500 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:45.304124 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:46.304212 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:46.304140 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:46.304657 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:46.304266 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:46.304657 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:46.304140 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:46.304788 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:46.304709 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:47.304715 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:47.304674 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:47.305212 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.304803 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:47.814281 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:47.814184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:47.814463 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.814305 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:47.814463 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.814397 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:55.814374818 +0000 UTC m=+18.069275004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:47.914589 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:47.914553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:47.914589 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:47.914593 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:47.914827 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.914701 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:47.914827 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.914717 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:16:47.914827 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.914737 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:16:47.914827 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.914750 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:47.914827 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.914760 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:55.914742308 +0000 UTC m=+18.169642491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:47.914827 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:47.914783 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:16:55.914773697 +0000 UTC m=+18.169673885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:48.305391 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:48.305309 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:48.305853 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:48.305418 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:48.305853 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:48.305457 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:48.305853 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:48.305637 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:49.304913 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:49.304881 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:49.305097 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:49.305011 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:50.304322 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:50.304287 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:50.304800 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:50.304423 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:50.304800 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:50.304576 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:50.304800 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:50.304671 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:51.304335 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:51.304248 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:51.304767 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:51.304382 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:52.307108 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:52.307018 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:52.307533 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:52.307137 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:52.307533 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:52.307199 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:52.307533 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:52.307293 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:53.304522 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:53.304487 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:53.304699 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:53.304607 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:54.304701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:54.304664 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:54.305169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:54.304677 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:54.305169 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:54.304807 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:54.305169 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:54.304886 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:55.304915 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:55.304882 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:55.305410 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.304982 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:55.874201 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:55.874172 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:55.874373 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.874292 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:55.874373 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.874350 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:17:11.874332457 +0000 UTC m=+34.129232645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:16:55.975433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:55.975397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:55.975433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:55.975445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:55.975673 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.975578 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:55.975673 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.975601 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:16:55.975673 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.975628 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:16:55.975673 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.975638 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:55.975673 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.975647 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:17:11.9756262 +0000 UTC m=+34.230526384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:16:55.975951 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:55.975685 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:17:11.975672701 +0000 UTC m=+34.230573103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:16:56.304319 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:56.304259 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:56.304450 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:56.304370 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:56.304450 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:56.304396 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:56.304550 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:56.304442 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:57.304329 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:57.304294 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:57.304720 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:57.304430 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:16:58.305551 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:58.305518 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:16:58.306014 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:58.305622 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:16:58.306014 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:58.305686 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:16:58.306014 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:58.305821 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:16:59.304773 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:16:59.304736 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:16:59.304946 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:16:59.304847 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:00.304325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.304069 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:00.304325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.304263 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:00.305061 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:00.304898 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:00.305061 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:00.305020 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:00.492701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.492673 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-495xf" event={"ID":"af3759db-0951-4fdd-8f5d-b5ec058cbb3a","Type":"ContainerStarted","Data":"4e7aa8290afcbe43a60c1ae93d3b413fe64515eb695491a383a9afcb1ba4662f"} Jul 15 16:17:00.494822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.494797 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"1e784c5b8ad7d38152920b334c4e478c99a4e01d1d046685849dd9952e280aed"} Jul 15 16:17:00.494934 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.494895 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"ea001a1d46bde906222c09ee766bb8c6b69c907ede1c3deec6b56fa83897fbfc"} Jul 15 16:17:00.496031 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.496010 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-78lgd" event={"ID":"f9437532-8ae5-4ded-b3c5-987dffce18fa","Type":"ContainerStarted","Data":"bc221926c744a3d6e156f96c8e260b94143034a061439e22bbd7ef5e09e52e21"} Jul 15 16:17:00.497289 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.497254 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcbg5" event={"ID":"7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9","Type":"ContainerStarted","Data":"e893aae73ce369f4bec0838dd26f7696d630b00b0a41b112ad497b085c311b89"} Jul 15 16:17:00.498745 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.498719 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgpcx" event={"ID":"479fd180-dffc-45e3-8741-284cf398e817","Type":"ContainerStarted","Data":"05d535676b6124c1382d34118195dab9bc1a9ff81eed10b1b490f498e86f0c72"} Jul 15 16:17:00.500206 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.500186 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-twdwz" event={"ID":"e1252ca3-d2d7-4e5c-8f94-e40a530bf5c0","Type":"ContainerStarted","Data":"63496cffda7d88e7aeb26c3e535e5fa9bbd823a1cb865e090608b9352059d481"} Jul 15 16:17:00.501760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.501707 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" event={"ID":"7220e675-998f-4d9a-8e39-e508eb158616","Type":"ContainerStarted","Data":"478a413ae9dff7c09d9fd1dec98fdc419b9aa10c7b4eb2009408249d41ae80bd"} Jul 15 16:17:00.503022 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.503002 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" event={"ID":"0220f44d9b57d66146d6849b5a148314","Type":"ContainerStarted","Data":"6c2d59976c56b96ad70e6ff07f83de23cdbfb868ba957f7237e08b878e3b495a"} Jul 15 16:17:00.504288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.504261 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerStarted","Data":"252e08da0dc38383aea41b8b71ad223aae75563a34cc0c2d29119f0eda024fe6"} Jul 15 16:17:00.565882 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.565823 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-495xf" podStartSLOduration=2.952806757 podStartE2EDuration="22.565809653s" totalImagesPullingTime="19.613002896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:00.565671874 +0000 UTC m=+22.820572080" watchObservedRunningTime="2026-07-15 16:17:00.565809653 +0000 UTC m=+22.820709859" Jul 15 16:17:00.565998 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.565919 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-51.ec2.internal" podStartSLOduration=21.565913309 podStartE2EDuration="21.565913309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:16:42.522287537 +0000 UTC m=+4.777187744" watchObservedRunningTime="2026-07-15 16:17:00.565913309 +0000 UTC m=+22.820813514" Jul 15 16:17:00.672888 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.672828 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pgpcx" podStartSLOduration=2.9704889850000002 podStartE2EDuration="22.672813942s" totalImagesPullingTime="19.702324957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:00.634733537 +0000 UTC m=+22.889633743" watchObservedRunningTime="2026-07-15 16:17:00.672813942 +0000 UTC m=+22.927714182" Jul 15 16:17:00.673007 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.672974 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rcbg5" podStartSLOduration=11.159757582 podStartE2EDuration="22.672966808s" totalImagesPullingTime="11.513209226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:00.672956231 +0000 UTC m=+22.927856455" watchObservedRunningTime="2026-07-15 16:17:00.672966808 +0000 UTC m=+22.927867014" Jul 15 16:17:00.844374 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.844239 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-51.ec2.internal" podStartSLOduration=21.844223236 podStartE2EDuration="21.844223236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:00.733254455 +0000 UTC m=+22.988154661" watchObservedRunningTime="2026-07-15 16:17:00.844223236 +0000 UTC m=+23.099123489" Jul 15 16:17:00.880370 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.880347 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rcbg5_7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9/konnectivity-agent/0.log" limit=1024 Jul 15 16:17:00.997521 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:00.997480 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-78lgd" podStartSLOduration=3.383117888 podStartE2EDuration="22.997464285s" totalImagesPullingTime="19.614346397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:00.900781884 +0000 UTC m=+23.155682089" watchObservedRunningTime="2026-07-15 16:17:00.997464285 +0000 UTC m=+23.252364490" Jul 15 16:17:01.185270 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.185238 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rcbg5_7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9/konnectivity-agent/0.log" limit=1024 Jul 15 16:17:01.303990 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.303962 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:01.304120 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:01.304058 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:01.319885 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.319850 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:17:01.508291 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.508199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5ln4l" event={"ID":"c9ffdfba-ae60-445a-96dd-30d45e9bf0f2","Type":"ContainerStarted","Data":"2d329ee748275b892a923e644c640f5a8dced9dfb4c8ea6795e2082a3dbec4c7"} Jul 15 16:17:01.509904 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.509861 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad2f5186-47e7-41a8-a6c3-91b8093440b3" containerID="252e08da0dc38383aea41b8b71ad223aae75563a34cc0c2d29119f0eda024fe6" exitCode=0 Jul 15 16:17:01.510024 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.509946 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerDied","Data":"252e08da0dc38383aea41b8b71ad223aae75563a34cc0c2d29119f0eda024fe6"} Jul 15 16:17:01.513313 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513271 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:17:01.513660 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513640 2560 generic.go:356] "Generic (PLEG): container finished" podID="5912f599-2ac6-4169-900f-e5576f6d6e67" containerID="1e784c5b8ad7d38152920b334c4e478c99a4e01d1d046685849dd9952e280aed" exitCode=1 Jul 15 16:17:01.513750 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513724 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerDied","Data":"1e784c5b8ad7d38152920b334c4e478c99a4e01d1d046685849dd9952e280aed"} Jul 15 16:17:01.513824 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513804 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"f7473f7dba471ed65c76a58f2c3d95705cd56cd49ef9cbd6b4412ea7003d983b"} Jul 15 16:17:01.513882 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513856 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"252f5c4420447f6a6c117652e96ada46f52150231d7d6a56257ef2100513a94e"} Jul 15 16:17:01.513938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513921 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"ed7eb11708f8f97259d6eeb06005986433b68275b166c0813bd269708f5f8254"} Jul 15 16:17:01.513992 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.513966 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"de372d80d03af80f5e2781a38a06c39d45a2813bf6f19861d43479a27b65c822"} Jul 15 16:17:01.561778 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.561722 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-twdwz" podStartSLOduration=3.933147816 podStartE2EDuration="23.561702557s" totalImagesPullingTime="19.628554741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:01.005298555 +0000 UTC m=+23.260198772" watchObservedRunningTime="2026-07-15 16:17:01.561702557 +0000 UTC m=+23.816602754" Jul 15 16:17:01.616250 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:01.616205 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5ln4l" podStartSLOduration=3.990781154 podStartE2EDuration="23.616188932s" totalImagesPullingTime="19.625407778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:01.563831557 +0000 UTC m=+23.818731782" watchObservedRunningTime="2026-07-15 16:17:01.616188932 +0000 UTC m=+23.871089137" Jul 15 16:17:02.303506 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:02.303298 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 16:17:02.304401 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:02.304379 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:02.304620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:02.304412 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:02.304620 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:02.304504 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:02.304744 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:02.304637 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:02.517926 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:02.517839 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" event={"ID":"7220e675-998f-4d9a-8e39-e508eb158616","Type":"ContainerStarted","Data":"76ee8b613adc09bac3b029d2bb39f60450978063c75fbf54a755053d3ac6bcdc"} Jul 15 16:17:03.206249 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:03.206152 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T16:17:02.303488231Z","UUID":"30c3d61f-91c0-4572-85d8-146f33cbb488","Handler":null,"Name":"","Endpoint":""} Jul 15 16:17:03.207826 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:03.207804 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 16:17:03.207977 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:03.207846 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 16:17:03.304459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:03.304437 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:03.304651 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:03.304548 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:03.522698 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:03.522675 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:17:03.523207 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:03.523103 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"50c8deea1493d403b4174a6efc0f5daeb13c8ac5e62bbad9da1309dde956e787"} Jul 15 16:17:04.083347 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.083319 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:17:04.083987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.083926 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:17:04.304194 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.304154 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:04.304194 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.304196 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:04.304416 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:04.304294 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:04.304481 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:04.304404 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:04.527713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.527638 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" event={"ID":"7220e675-998f-4d9a-8e39-e508eb158616","Type":"ContainerStarted","Data":"9364520d6b7914fbd09484beb9ac777e2233922f70e907fc23ef05985238d583"} Jul 15 16:17:04.528523 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.528274 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rcbg5" Jul 15 16:17:04.603420 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:04.603371 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-njt2p" podStartSLOduration=3.964599862 podStartE2EDuration="26.603354281s" totalImagesPullingTime="22.638754419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:04.581007793 +0000 UTC m=+26.835908018" watchObservedRunningTime="2026-07-15 16:17:04.603354281 +0000 UTC m=+26.858254465" Jul 15 16:17:05.304939 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:05.304909 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:05.305105 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:05.305040 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:06.304284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:06.304187 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:06.304284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:06.304203 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:06.304997 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:06.304346 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:06.304997 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:06.304500 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:07.304961 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.304723 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:07.305273 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:07.304991 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:07.536403 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.536365 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad2f5186-47e7-41a8-a6c3-91b8093440b3" containerID="55708fd010217f1053ad8e41b811259101ea2dbf2a8a6be4beb547cdca687002" exitCode=0 Jul 15 16:17:07.536579 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.536456 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerDied","Data":"55708fd010217f1053ad8e41b811259101ea2dbf2a8a6be4beb547cdca687002"} Jul 15 16:17:07.542043 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.542013 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:17:07.542458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.542428 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"d94de18831da260e2cb84d71edc5d4c238b2bd45ab6f40f3407336beeb24c3eb"} Jul 15 16:17:07.542658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.542636 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:17:07.542761 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.542708 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:17:07.542839 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.542825 2560 scope.go:117] "RemoveContainer" containerID="1e784c5b8ad7d38152920b334c4e478c99a4e01d1d046685849dd9952e280aed" Jul 15 16:17:07.558579 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:07.558529 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:17:08.304926 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.304895 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:08.305183 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:08.304989 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:08.305183 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.305040 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:08.305183 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:08.305125 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:08.547074 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.547050 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:17:08.547430 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.547406 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" event={"ID":"5912f599-2ac6-4169-900f-e5576f6d6e67","Type":"ContainerStarted","Data":"e6014b1fc8a649a578accd550d2c2a5032d34bbceac8bd3b59af5752ce4d5592"} Jul 15 16:17:08.547571 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.547557 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:17:08.562589 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.562539 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:17:08.656619 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:08.656579 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" podStartSLOduration=10.962891814 podStartE2EDuration="30.656567732s" totalImagesPullingTime="19.693675918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:08.614400566 +0000 UTC m=+30.869300774" watchObservedRunningTime="2026-07-15 16:17:08.656567732 +0000 UTC m=+30.911467954" Jul 15 16:17:09.304642 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:09.304472 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:09.304792 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:09.304707 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:10.304030 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:10.303953 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:10.304446 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:10.304061 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:10.304446 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:10.304115 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:10.304446 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:10.304234 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:10.552947 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:10.552914 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad2f5186-47e7-41a8-a6c3-91b8093440b3" containerID="86c330a435761077e127c89c40b7f8c87bc31850ebc1bc226113ed0f23b4756e" exitCode=0 Jul 15 16:17:10.553106 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:10.552999 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerDied","Data":"86c330a435761077e127c89c40b7f8c87bc31850ebc1bc226113ed0f23b4756e"} Jul 15 16:17:11.304773 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:11.304747 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:11.305195 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.304891 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:11.557589 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:11.557512 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad2f5186-47e7-41a8-a6c3-91b8093440b3" containerID="c7327c72a07666ccb0ce9b435dee9596bf8fe73e8180228be247bbbf62b07150" exitCode=0 Jul 15 16:17:11.557589 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:11.557549 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerDied","Data":"c7327c72a07666ccb0ce9b435dee9596bf8fe73e8180228be247bbbf62b07150"} Jul 15 16:17:11.898614 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:11.898533 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:11.898750 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.898665 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 16:17:11.898750 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.898724 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret podName:7b77867d-4145-4515-945f-7a50589dd046 nodeName:}" failed. No retries permitted until 2026-07-15 16:17:43.898705386 +0000 UTC m=+66.153605571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret") pod "global-pull-secret-syncer-ks4kq" (UID: "7b77867d-4145-4515-945f-7a50589dd046") : object "kube-system"/"original-pull-secret" not registered Jul 15 16:17:11.999318 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:11.999290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:11.999479 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:11.999324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:11.999479 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.999448 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 16:17:11.999479 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.999455 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:17:11.999479 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.999464 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 16:17:11.999479 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.999474 2560 projected.go:196] Error preparing data for projected volume kube-api-access-98wm4 for pod openshift-network-diagnostics/network-check-target-cf9rh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:17:11.999728 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.999513 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs podName:8bed2434-1dfa-46d3-908f-b9a243db93a2 nodeName:}" failed. No retries permitted until 2026-07-15 16:17:43.999494943 +0000 UTC m=+66.254395128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs") pod "network-metrics-daemon-flp4t" (UID: "8bed2434-1dfa-46d3-908f-b9a243db93a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 16:17:11.999728 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:11.999532 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4 podName:2e0d546f-57ef-450e-920e-e707ea8876e4 nodeName:}" failed. No retries permitted until 2026-07-15 16:17:43.999522114 +0000 UTC m=+66.254422297 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-98wm4" (UniqueName: "kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4") pod "network-check-target-cf9rh" (UID: "2e0d546f-57ef-450e-920e-e707ea8876e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 16:17:12.304936 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:12.304844 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:12.305346 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:12.304992 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:12.305346 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:12.305051 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:12.305346 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:12.305180 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:13.304179 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:13.304146 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:13.304360 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:13.304268 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:14.304116 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:14.304080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:14.304526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:14.304086 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:14.304526 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:14.304200 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:14.304526 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:14.304285 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:15.304815 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:15.304779 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:15.305247 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:15.304929 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:16.303996 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:16.303970 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:16.303996 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:16.303988 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:16.304198 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:16.304066 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:16.304198 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:16.304188 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:16.570679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:16.570585 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad2f5186-47e7-41a8-a6c3-91b8093440b3" containerID="b99f4d8a013455ebeda96b6faf700415c5f054923c6394eaf37c24558fb5a2a9" exitCode=0 Jul 15 16:17:16.570679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:16.570652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerDied","Data":"b99f4d8a013455ebeda96b6faf700415c5f054923c6394eaf37c24558fb5a2a9"} Jul 15 16:17:17.304019 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:17.303990 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:17.304180 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:17.304093 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:17.576093 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:17.576012 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad2f5186-47e7-41a8-a6c3-91b8093440b3" containerID="6845edc0c2131da1195a6ca0bdf0e6bf510efefe5ad258676fdec885923c3480" exitCode=0 Jul 15 16:17:17.576093 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:17.576069 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerDied","Data":"6845edc0c2131da1195a6ca0bdf0e6bf510efefe5ad258676fdec885923c3480"} Jul 15 16:17:18.305309 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:18.305276 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:18.305469 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:18.305352 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:18.305469 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:18.305455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:18.305594 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:18.305572 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:18.582103 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:18.582033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" event={"ID":"ad2f5186-47e7-41a8-a6c3-91b8093440b3","Type":"ContainerStarted","Data":"ec51c4f0a233f87e77b0dd3ac57dcfef79d5b7cf69ab534226c9c3bd594f73e7"} Jul 15 16:17:18.616931 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:18.616894 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vtfzn" podStartSLOduration=8.56504692 podStartE2EDuration="40.616859705s" totalImagesPullingTime="32.051812785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:18.616346089 +0000 UTC m=+40.871246305" watchObservedRunningTime="2026-07-15 16:17:18.616859705 +0000 UTC m=+40.871759911" Jul 15 16:17:19.303910 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:19.303885 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:19.304068 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:19.303980 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:20.304014 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:20.303981 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:20.304489 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:20.303988 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:20.304489 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:20.304101 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:20.304489 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:20.304154 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:21.304067 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:21.304036 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:21.304459 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:21.304133 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:22.304280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:22.304251 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:22.304626 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:22.304346 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:22.304626 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:22.304422 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:22.304626 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:22.304503 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:23.304733 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.304708 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:23.305252 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:23.304806 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:23.832490 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.832237 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ks4kq"] Jul 15 16:17:23.832762 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.832744 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cf9rh"] Jul 15 16:17:23.833017 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.832999 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:23.833272 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:23.833246 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:23.833674 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.833653 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:23.834238 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:23.834215 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:23.837531 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.837511 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flp4t"] Jul 15 16:17:23.837606 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:23.837590 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:23.837679 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:23.837664 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:25.303950 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:25.303924 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:25.303950 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:25.303954 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:25.304365 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:25.304024 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:25.304365 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:25.304073 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:26.304802 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:26.304771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:26.305193 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:26.304927 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flp4t" podUID="8bed2434-1dfa-46d3-908f-b9a243db93a2" Jul 15 16:17:27.304726 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:27.304693 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:27.304915 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:27.304693 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:27.304915 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:27.304809 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cf9rh" podUID="2e0d546f-57ef-450e-920e-e707ea8876e4" Jul 15 16:17:27.304915 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:27.304862 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks4kq" podUID="7b77867d-4145-4515-945f-7a50589dd046" Jul 15 16:17:28.046640 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.046567 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-51.ec2.internal" event="NodeReady" Jul 15 16:17:28.046775 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.046679 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 16:17:28.099763 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.099735 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7rttt"] Jul 15 16:17:28.111241 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.111219 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x4fdz"] Jul 15 16:17:28.111368 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.111331 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.114297 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.114243 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vkb7c\"" Jul 15 16:17:28.114979 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.114758 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 16:17:28.117742 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.115309 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 16:17:28.123620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.123603 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rttt"] Jul 15 16:17:28.123620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.123631 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4fdz"] Jul 15 16:17:28.123787 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.123739 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.127041 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.127017 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 16:17:28.127205 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.127057 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tw9wx\"" Jul 15 16:17:28.127205 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.127076 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 16:17:28.127205 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.127057 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 16:17:28.127425 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.127412 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 16:17:28.131722 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.131705 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hc5rh"] Jul 15 16:17:28.150363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.150343 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hc5rh"] Jul 15 16:17:28.150471 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.150433 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.153322 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.153304 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 16:17:28.153472 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.153454 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 16:17:28.153811 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.153798 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 16:17:28.153861 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.153839 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7vwjh\"" Jul 15 16:17:28.226024 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.225998 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96c0b345-1601-439b-81ba-940e5e8712b4-data-volume\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.226145 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226029 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75e64c35-a318-46b4-857b-46b61dbf3e6f-cert\") pod \"ingress-canary-hc5rh\" (UID: \"75e64c35-a318-46b4-857b-46b61dbf3e6f\") " pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.226145 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226054 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4sx\" (UniqueName: \"kubernetes.io/projected/12f6fde1-e731-482e-b77c-8d19635faa93-kube-api-access-hn4sx\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.226145 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226080 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96c0b345-1601-439b-81ba-940e5e8712b4-crio-socket\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.226254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226142 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12f6fde1-e731-482e-b77c-8d19635faa93-metrics-tls\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.226254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12f6fde1-e731-482e-b77c-8d19635faa93-config-volume\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.226254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226220 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f6fde1-e731-482e-b77c-8d19635faa93-tmp-dir\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.226254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226239 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96c0b345-1601-439b-81ba-940e5e8712b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.226367 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226255 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96c0b345-1601-439b-81ba-940e5e8712b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.226367 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226273 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpzb\" (UniqueName: \"kubernetes.io/projected/75e64c35-a318-46b4-857b-46b61dbf3e6f-kube-api-access-crpzb\") pod \"ingress-canary-hc5rh\" (UID: \"75e64c35-a318-46b4-857b-46b61dbf3e6f\") " pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.226367 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.226289 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzfw\" (UniqueName: \"kubernetes.io/projected/96c0b345-1601-439b-81ba-940e5e8712b4-kube-api-access-nvzfw\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.305547 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.305475 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:28.308387 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.308369 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wsr6k\"" Jul 15 16:17:28.308483 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.308390 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 16:17:28.327443 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327418 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96c0b345-1601-439b-81ba-940e5e8712b4-data-volume\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.327537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327447 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75e64c35-a318-46b4-857b-46b61dbf3e6f-cert\") pod \"ingress-canary-hc5rh\" (UID: \"75e64c35-a318-46b4-857b-46b61dbf3e6f\") " pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.327537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327465 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4sx\" (UniqueName: \"kubernetes.io/projected/12f6fde1-e731-482e-b77c-8d19635faa93-kube-api-access-hn4sx\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.327537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96c0b345-1601-439b-81ba-940e5e8712b4-crio-socket\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.327537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327523 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12f6fde1-e731-482e-b77c-8d19635faa93-metrics-tls\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.327713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327552 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12f6fde1-e731-482e-b77c-8d19635faa93-config-volume\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.327713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f6fde1-e731-482e-b77c-8d19635faa93-tmp-dir\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.327713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327636 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96c0b345-1601-439b-81ba-940e5e8712b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.327713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96c0b345-1601-439b-81ba-940e5e8712b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.327713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpzb\" (UniqueName: \"kubernetes.io/projected/75e64c35-a318-46b4-857b-46b61dbf3e6f-kube-api-access-crpzb\") pod \"ingress-canary-hc5rh\" (UID: \"75e64c35-a318-46b4-857b-46b61dbf3e6f\") " pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.327952 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327726 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzfw\" (UniqueName: \"kubernetes.io/projected/96c0b345-1601-439b-81ba-940e5e8712b4-kube-api-access-nvzfw\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.327952 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327777 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96c0b345-1601-439b-81ba-940e5e8712b4-data-volume\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.328052 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.327965 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96c0b345-1601-439b-81ba-940e5e8712b4-crio-socket\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.328299 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.328273 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96c0b345-1601-439b-81ba-940e5e8712b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.328381 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.328273 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12f6fde1-e731-482e-b77c-8d19635faa93-config-volume\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.331481 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.331457 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12f6fde1-e731-482e-b77c-8d19635faa93-metrics-tls\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.331565 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.331497 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96c0b345-1601-439b-81ba-940e5e8712b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.331673 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.331660 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75e64c35-a318-46b4-857b-46b61dbf3e6f-cert\") pod \"ingress-canary-hc5rh\" (UID: \"75e64c35-a318-46b4-857b-46b61dbf3e6f\") " pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.340202 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.340178 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f6fde1-e731-482e-b77c-8d19635faa93-tmp-dir\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.340278 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.340250 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzfw\" (UniqueName: \"kubernetes.io/projected/96c0b345-1601-439b-81ba-940e5e8712b4-kube-api-access-nvzfw\") pod \"insights-runtime-extractor-x4fdz\" (UID: \"96c0b345-1601-439b-81ba-940e5e8712b4\") " pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.341242 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.341222 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpzb\" (UniqueName: \"kubernetes.io/projected/75e64c35-a318-46b4-857b-46b61dbf3e6f-kube-api-access-crpzb\") pod \"ingress-canary-hc5rh\" (UID: \"75e64c35-a318-46b4-857b-46b61dbf3e6f\") " pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.341710 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.341691 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4sx\" (UniqueName: \"kubernetes.io/projected/12f6fde1-e731-482e-b77c-8d19635faa93-kube-api-access-hn4sx\") pod \"dns-default-7rttt\" (UID: \"12f6fde1-e731-482e-b77c-8d19635faa93\") " pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.424914 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.424890 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:28.432163 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.432141 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4fdz" Jul 15 16:17:28.457676 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.457656 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hc5rh" Jul 15 16:17:28.614108 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.613940 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hc5rh"] Jul 15 16:17:28.617763 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.617735 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4fdz"] Jul 15 16:17:28.618541 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:28.618519 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rttt"] Jul 15 16:17:28.618921 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:28.618896 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e64c35_a318_46b4_857b_46b61dbf3e6f.slice/crio-d013b0e98a331129a97b64c871f5aa7dded4c3fa20a11b7b17faf8cdbfe2487e WatchSource:0}: Error finding container d013b0e98a331129a97b64c871f5aa7dded4c3fa20a11b7b17faf8cdbfe2487e: Status 404 returned error can't find the container with id d013b0e98a331129a97b64c871f5aa7dded4c3fa20a11b7b17faf8cdbfe2487e Jul 15 16:17:28.621683 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:28.621662 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f6fde1_e731_482e_b77c_8d19635faa93.slice/crio-fb85d6382b3f88cbb9b091f0fb63635bbc47d93b5f9ff4a808388052ffcaf268 WatchSource:0}: Error finding container fb85d6382b3f88cbb9b091f0fb63635bbc47d93b5f9ff4a808388052ffcaf268: Status 404 returned error can't find the container with id fb85d6382b3f88cbb9b091f0fb63635bbc47d93b5f9ff4a808388052ffcaf268 Jul 15 16:17:28.622538 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:28.622509 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c0b345_1601_439b_81ba_940e5e8712b4.slice/crio-ebdc0fc85893bd5c455ab26afc15163b226e11ca72c351e3aee5fa67d1dd81f5 WatchSource:0}: Error finding container ebdc0fc85893bd5c455ab26afc15163b226e11ca72c351e3aee5fa67d1dd81f5: Status 404 returned error can't find the container with id ebdc0fc85893bd5c455ab26afc15163b226e11ca72c351e3aee5fa67d1dd81f5 Jul 15 16:17:29.304062 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.304034 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:29.304281 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.304034 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:29.307249 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.307227 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 16:17:29.308402 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.308380 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 16:17:29.308668 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.308651 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 16:17:29.308803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.308785 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dmv8w\"" Jul 15 16:17:29.607103 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.607004 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rttt" event={"ID":"12f6fde1-e731-482e-b77c-8d19635faa93","Type":"ContainerStarted","Data":"fb85d6382b3f88cbb9b091f0fb63635bbc47d93b5f9ff4a808388052ffcaf268"} Jul 15 16:17:29.612085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.609198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hc5rh" event={"ID":"75e64c35-a318-46b4-857b-46b61dbf3e6f","Type":"ContainerStarted","Data":"d013b0e98a331129a97b64c871f5aa7dded4c3fa20a11b7b17faf8cdbfe2487e"} Jul 15 16:17:29.612085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:29.610428 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4fdz" event={"ID":"96c0b345-1601-439b-81ba-940e5e8712b4","Type":"ContainerStarted","Data":"ebdc0fc85893bd5c455ab26afc15163b226e11ca72c351e3aee5fa67d1dd81f5"} Jul 15 16:17:30.644518 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.644485 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ttl9v"] Jul 15 16:17:30.663864 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.663834 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ttl9v"] Jul 15 16:17:30.664036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.663975 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.667155 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.667132 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tkw2w\"" Jul 15 16:17:30.668637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.668338 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 16:17:30.668637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.668371 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 16:17:30.668637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.668340 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 16:17:30.668637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.668343 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 16:17:30.668637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.668604 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 16:17:30.749977 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.749949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42c103cd-610d-49d1-8a0a-a652c7d9261d-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.750136 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.749985 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2c6v\" (UniqueName: \"kubernetes.io/projected/42c103cd-610d-49d1-8a0a-a652c7d9261d-kube-api-access-j2c6v\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.750136 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.750025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42c103cd-610d-49d1-8a0a-a652c7d9261d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.750136 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.750117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42c103cd-610d-49d1-8a0a-a652c7d9261d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.851595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.851476 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42c103cd-610d-49d1-8a0a-a652c7d9261d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.851595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.851522 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42c103cd-610d-49d1-8a0a-a652c7d9261d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.851595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.851605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42c103cd-610d-49d1-8a0a-a652c7d9261d-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.851904 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.851630 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2c6v\" (UniqueName: \"kubernetes.io/projected/42c103cd-610d-49d1-8a0a-a652c7d9261d-kube-api-access-j2c6v\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.852692 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.852662 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42c103cd-610d-49d1-8a0a-a652c7d9261d-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.855759 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.855735 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42c103cd-610d-49d1-8a0a-a652c7d9261d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.856149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.856073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/42c103cd-610d-49d1-8a0a-a652c7d9261d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.860710 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.860688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2c6v\" (UniqueName: \"kubernetes.io/projected/42c103cd-610d-49d1-8a0a-a652c7d9261d-kube-api-access-j2c6v\") pod \"prometheus-operator-64785b6675-ttl9v\" (UID: \"42c103cd-610d-49d1-8a0a-a652c7d9261d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:30.976478 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:30.976395 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" Jul 15 16:17:34.724609 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:34.724381 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ttl9v"] Jul 15 16:17:34.729141 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:34.729100 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c103cd_610d_49d1_8a0a_a652c7d9261d.slice/crio-a70ca7909f28a44bfb664c84029772c31dfa9b4e47febb136e464312f804a639 WatchSource:0}: Error finding container a70ca7909f28a44bfb664c84029772c31dfa9b4e47febb136e464312f804a639: Status 404 returned error can't find the container with id a70ca7909f28a44bfb664c84029772c31dfa9b4e47febb136e464312f804a639 Jul 15 16:17:35.630387 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.630351 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hc5rh" event={"ID":"75e64c35-a318-46b4-857b-46b61dbf3e6f","Type":"ContainerStarted","Data":"fbaf72dedeced84fbd1f0f1001305a069795ea33e453e9668fede5e4acd802fd"} Jul 15 16:17:35.631914 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.631894 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4fdz" event={"ID":"96c0b345-1601-439b-81ba-940e5e8712b4","Type":"ContainerStarted","Data":"9fe1fe2f6e7cf429da69fa72eb74d80a5e06e00ccdfe3da1822b5385d51615e0"} Jul 15 16:17:35.633419 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.633398 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rttt" event={"ID":"12f6fde1-e731-482e-b77c-8d19635faa93","Type":"ContainerStarted","Data":"885f9db4b3918214aacff7e44bdfcee409d1884573c30c0a88401c910d02c8c2"} Jul 15 16:17:35.633526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.633447 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:35.633526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.633471 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rttt" event={"ID":"12f6fde1-e731-482e-b77c-8d19635faa93","Type":"ContainerStarted","Data":"46d52b3acdceee582efdc1f818643f5e954f8d958f5b73f61f6e1a122b040a4b"} Jul 15 16:17:35.634483 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.634463 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" event={"ID":"42c103cd-610d-49d1-8a0a-a652c7d9261d","Type":"ContainerStarted","Data":"a70ca7909f28a44bfb664c84029772c31dfa9b4e47febb136e464312f804a639"} Jul 15 16:17:35.676257 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:35.676219 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hc5rh" podStartSLOduration=1.709271235 podStartE2EDuration="7.676206542s" totalImagesPullingTime="5.966935307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:35.676156501 +0000 UTC m=+57.931056720" watchObservedRunningTime="2026-07-15 16:17:35.676206542 +0000 UTC m=+57.931106726" Jul 15 16:17:36.641359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:36.640932 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" event={"ID":"42c103cd-610d-49d1-8a0a-a652c7d9261d","Type":"ContainerStarted","Data":"44b0e83167aa8740cfc7af6c943dae5b02818b17979c0ed5cd129fdc036726e3"} Jul 15 16:17:37.647216 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:37.647176 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" event={"ID":"42c103cd-610d-49d1-8a0a-a652c7d9261d","Type":"ContainerStarted","Data":"73c6605f7ddac3e2357458e2a03c310fca862badf27a9f6c2eb7f9758ac8b6a3"} Jul 15 16:17:37.666823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:37.666764 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-ttl9v" podStartSLOduration=5.854350329 podStartE2EDuration="7.666740959s" totalImagesPullingTime="1.81239063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:37.665816134 +0000 UTC m=+59.920716373" watchObservedRunningTime="2026-07-15 16:17:37.666740959 +0000 UTC m=+59.921641166" Jul 15 16:17:37.667303 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:37.667275 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7rttt" podStartSLOduration=4.023147899 podStartE2EDuration="9.667265341s" totalImagesPullingTime="5.644117442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:35.71632655 +0000 UTC m=+57.971226758" watchObservedRunningTime="2026-07-15 16:17:37.667265341 +0000 UTC m=+59.922165547" Jul 15 16:17:39.655537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:39.655506 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4fdz" event={"ID":"96c0b345-1601-439b-81ba-940e5e8712b4","Type":"ContainerStarted","Data":"c07c8b4bf14e190432b62df57c0b512c56e268760bdc84a6717f7f1746ed3566"} Jul 15 16:17:40.248642 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.248608 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d"] Jul 15 16:17:40.251235 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.251219 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.254809 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.254788 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 16:17:40.254941 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.254832 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 16:17:40.254941 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.254837 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wh799\"" Jul 15 16:17:40.271342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.271322 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d"] Jul 15 16:17:40.321450 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.321428 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3041f42a-b504-4229-ae91-038d98c9bae2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.321541 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.321459 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbrv\" (UniqueName: \"kubernetes.io/projected/3041f42a-b504-4229-ae91-038d98c9bae2-kube-api-access-cvbrv\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.321585 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.321563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3041f42a-b504-4229-ae91-038d98c9bae2-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.321629 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.321606 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3041f42a-b504-4229-ae91-038d98c9bae2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.339110 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.339091 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hjblt"] Jul 15 16:17:40.341499 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.341482 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.346431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.346408 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 16:17:40.346701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.346620 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 16:17:40.347960 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.347942 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 16:17:40.349909 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.349895 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6pvz6\"" Jul 15 16:17:40.422389 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422362 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422522 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422408 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3041f42a-b504-4229-ae91-038d98c9bae2-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.422522 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422480 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99bfa9e6-40d0-4685-925f-49493b61683b-metrics-client-ca\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422522 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422517 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-root\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422631 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422533 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-textfile\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422631 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422560 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422631 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422581 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-sys\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422744 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3041f42a-b504-4229-ae91-038d98c9bae2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.422744 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422744 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422722 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3041f42a-b504-4229-ae91-038d98c9bae2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.422897 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbrv\" (UniqueName: \"kubernetes.io/projected/3041f42a-b504-4229-ae91-038d98c9bae2-kube-api-access-cvbrv\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.422965 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhf4d\" (UniqueName: \"kubernetes.io/projected/99bfa9e6-40d0-4685-925f-49493b61683b-kube-api-access-fhf4d\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.422965 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.422937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-wtmp\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.423109 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.423090 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3041f42a-b504-4229-ae91-038d98c9bae2-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.425209 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.425192 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3041f42a-b504-4229-ae91-038d98c9bae2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.425270 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.425206 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3041f42a-b504-4229-ae91-038d98c9bae2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.432863 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.432842 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbrv\" (UniqueName: \"kubernetes.io/projected/3041f42a-b504-4229-ae91-038d98c9bae2-kube-api-access-cvbrv\") pod \"openshift-state-metrics-84bd68b896-jqj6d\" (UID: \"3041f42a-b504-4229-ae91-038d98c9bae2\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.523455 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523455 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99bfa9e6-40d0-4685-925f-49493b61683b-metrics-client-ca\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523469 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-root\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523507 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-root\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523528 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-textfile\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523555 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523578 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-sys\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523596 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523624 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf4d\" (UniqueName: \"kubernetes.io/projected/99bfa9e6-40d0-4685-925f-49493b61683b-kube-api-access-fhf4d\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-wtmp\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523702 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-sys\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523777 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-wtmp\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:40.523828 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:40.523925 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls podName:99bfa9e6-40d0-4685-925f-49493b61683b nodeName:}" failed. No retries permitted until 2026-07-15 16:17:41.02390506 +0000 UTC m=+63.278805252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls") pod "node-exporter-hjblt" (UID: "99bfa9e6-40d0-4685-925f-49493b61683b") : secret "node-exporter-tls" not found Jul 15 16:17:40.523938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.523921 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-textfile\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.524243 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.524024 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.524243 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.524111 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99bfa9e6-40d0-4685-925f-49493b61683b-metrics-client-ca\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.525602 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.525584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.533748 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.533727 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf4d\" (UniqueName: \"kubernetes.io/projected/99bfa9e6-40d0-4685-925f-49493b61683b-kube-api-access-fhf4d\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:40.559068 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.559043 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" Jul 15 16:17:40.563341 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.563326 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-65mm5" Jul 15 16:17:40.696927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:40.696899 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d"] Jul 15 16:17:40.700047 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:40.700019 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3041f42a_b504_4229_ae91_038d98c9bae2.slice/crio-7d4113f8a2b8ac9918eb832c7fcf276d19cf819476fff0e5633ba4e1fffe9c91 WatchSource:0}: Error finding container 7d4113f8a2b8ac9918eb832c7fcf276d19cf819476fff0e5633ba4e1fffe9c91: Status 404 returned error can't find the container with id 7d4113f8a2b8ac9918eb832c7fcf276d19cf819476fff0e5633ba4e1fffe9c91 Jul 15 16:17:41.027545 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.027460 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:41.027678 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:41.027607 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 16:17:41.027678 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:17:41.027669 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls podName:99bfa9e6-40d0-4685-925f-49493b61683b nodeName:}" failed. No retries permitted until 2026-07-15 16:17:42.027651608 +0000 UTC m=+64.282551809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls") pod "node-exporter-hjblt" (UID: "99bfa9e6-40d0-4685-925f-49493b61683b") : secret "node-exporter-tls" not found Jul 15 16:17:41.299051 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.298967 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:17:41.301430 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.301415 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.305091 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305070 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 16:17:41.305209 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305140 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 16:17:41.305209 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305157 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 16:17:41.305209 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305157 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 16:17:41.305363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305247 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 16:17:41.305363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305328 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 16:17:41.305468 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305448 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 16:17:41.305574 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305559 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9m695\"" Jul 15 16:17:41.305638 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305596 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 16:17:41.305906 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.305891 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 16:17:41.339441 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.339416 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:17:41.431361 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431361 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431362 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431385 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431405 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431536 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcm4f\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-kube-api-access-dcm4f\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431573 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-config-out\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431684 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-web-config\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.431804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.431738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532299 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532262 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532515 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532515 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532515 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532515 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532391 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-web-config\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532515 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532451 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.532515 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532508 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532531 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532566 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcm4f\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-kube-api-access-dcm4f\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.532711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-config-out\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.533791 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.533059 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.534682 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.534657 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.534837 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.534725 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.535393 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.535369 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-config-out\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.536626 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.536581 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.537482 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.537457 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.537853 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.537810 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-web-config\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.538189 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.538166 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.538316 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.538209 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.538475 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.538437 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.538797 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.538773 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.538912 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.538790 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.546424 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.546400 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcm4f\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-kube-api-access-dcm4f\") pod \"alertmanager-main-0\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.611280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.611253 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:17:41.667073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.667011 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" event={"ID":"3041f42a-b504-4229-ae91-038d98c9bae2","Type":"ContainerStarted","Data":"c816341ab41b47a1ff0c9116c99a02004129e65829d7f10fc0fab8d5e0aabdff"} Jul 15 16:17:41.667210 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.667120 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" event={"ID":"3041f42a-b504-4229-ae91-038d98c9bae2","Type":"ContainerStarted","Data":"1ee98dc57680af853565a8280e148ee413f0a1a7f3d92afe412a3e5b3fd4059b"} Jul 15 16:17:41.667210 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.667156 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" event={"ID":"3041f42a-b504-4229-ae91-038d98c9bae2","Type":"ContainerStarted","Data":"7d4113f8a2b8ac9918eb832c7fcf276d19cf819476fff0e5633ba4e1fffe9c91"} Jul 15 16:17:41.773511 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:41.773479 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:17:41.777323 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:41.777294 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3709ae13_90a7_4f38_a159_5e55fadc963b.slice/crio-48f49d3ac972ae7190563802f655acad4f823e2d6093f80a8f91727cb2bc48d6 WatchSource:0}: Error finding container 48f49d3ac972ae7190563802f655acad4f823e2d6093f80a8f91727cb2bc48d6: Status 404 returned error can't find the container with id 48f49d3ac972ae7190563802f655acad4f823e2d6093f80a8f91727cb2bc48d6 Jul 15 16:17:42.036573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.036463 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:42.039325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.039289 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99bfa9e6-40d0-4685-925f-49493b61683b-node-exporter-tls\") pod \"node-exporter-hjblt\" (UID: \"99bfa9e6-40d0-4685-925f-49493b61683b\") " pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:42.148909 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.148881 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjblt" Jul 15 16:17:42.159860 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:42.159829 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bfa9e6_40d0_4685_925f_49493b61683b.slice/crio-f1f662fc33caf6196ad1ad14f878e568d246838de79e9c2d74dabbf317fe3c61 WatchSource:0}: Error finding container f1f662fc33caf6196ad1ad14f878e568d246838de79e9c2d74dabbf317fe3c61: Status 404 returned error can't find the container with id f1f662fc33caf6196ad1ad14f878e568d246838de79e9c2d74dabbf317fe3c61 Jul 15 16:17:42.303832 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.303747 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk"] Jul 15 16:17:42.307017 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.306989 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.310301 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.310129 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 16:17:42.310301 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.310153 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 16:17:42.310301 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.310131 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 16:17:42.311066 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.311044 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lcxms\"" Jul 15 16:17:42.311189 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.311096 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 16:17:42.311609 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.311335 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 16:17:42.311609 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.311384 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2iq3gq2f9fpog\"" Jul 15 16:17:42.322715 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.322696 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk"] Jul 15 16:17:42.440929 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.440848 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-tls\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.440960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-grpc-tls\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.440987 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.441024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/4eca1df6-3d6d-480b-bbee-abfb87e4bece-kube-api-access-6kckp\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441265 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.441170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441265 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.441216 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441380 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.441275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.441380 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.441350 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca1df6-3d6d-480b-bbee-abfb87e4bece-metrics-client-ca\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542514 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542480 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542613 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542546 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542613 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542613 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca1df6-3d6d-480b-bbee-abfb87e4bece-metrics-client-ca\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542773 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-tls\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542773 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542750 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-grpc-tls\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542852 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542783 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.542852 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.542820 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/4eca1df6-3d6d-480b-bbee-abfb87e4bece-kube-api-access-6kckp\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.543821 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.543773 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca1df6-3d6d-480b-bbee-abfb87e4bece-metrics-client-ca\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.546391 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.546348 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-tls\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.547139 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.546502 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.547139 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.547015 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-grpc-tls\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.548435 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.548387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.548690 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.548654 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.549160 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.549135 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4eca1df6-3d6d-480b-bbee-abfb87e4bece-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.552460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.552437 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/4eca1df6-3d6d-480b-bbee-abfb87e4bece-kube-api-access-6kckp\") pod \"thanos-querier-7d8f5586cc-gfpkk\" (UID: \"4eca1df6-3d6d-480b-bbee-abfb87e4bece\") " pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.619218 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.619183 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:42.672506 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.672468 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjblt" event={"ID":"99bfa9e6-40d0-4685-925f-49493b61683b","Type":"ContainerStarted","Data":"f1f662fc33caf6196ad1ad14f878e568d246838de79e9c2d74dabbf317fe3c61"} Jul 15 16:17:42.675556 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.675506 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" event={"ID":"3041f42a-b504-4229-ae91-038d98c9bae2","Type":"ContainerStarted","Data":"158e2350d1499603c602f64665352cf7430a22c4b054ce0bb31d3bc6b84e7e03"} Jul 15 16:17:42.677588 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.677508 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"48f49d3ac972ae7190563802f655acad4f823e2d6093f80a8f91727cb2bc48d6"} Jul 15 16:17:42.698638 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.698577 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jqj6d" podStartSLOduration=1.032227085 podStartE2EDuration="2.698553834s" totalImagesPullingTime="1.666326749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:42.697279458 +0000 UTC m=+64.952179665" watchObservedRunningTime="2026-07-15 16:17:42.698553834 +0000 UTC m=+64.953454042" Jul 15 16:17:42.781360 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:42.781264 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk"] Jul 15 16:17:42.784008 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:42.783977 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eca1df6_3d6d_480b_bbee_abfb87e4bece.slice/crio-857b1f61975c9eb58e18efc29acc6ecf769878121d1e3c1f7c8d9fc14c29f76d WatchSource:0}: Error finding container 857b1f61975c9eb58e18efc29acc6ecf769878121d1e3c1f7c8d9fc14c29f76d: Status 404 returned error can't find the container with id 857b1f61975c9eb58e18efc29acc6ecf769878121d1e3c1f7c8d9fc14c29f76d Jul 15 16:17:43.682166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:43.682134 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"857b1f61975c9eb58e18efc29acc6ecf769878121d1e3c1f7c8d9fc14c29f76d"} Jul 15 16:17:43.958575 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:43.958494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:43.961955 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:43.961926 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 16:17:43.971863 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:43.971833 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b77867d-4145-4515-945f-7a50589dd046-original-pull-secret\") pod \"global-pull-secret-syncer-ks4kq\" (UID: \"7b77867d-4145-4515-945f-7a50589dd046\") " pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:44.018491 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.018467 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks4kq" Jul 15 16:17:44.059474 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.059449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:44.059591 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.059487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:44.062511 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.062490 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 16:17:44.063310 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.063286 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 16:17:44.072930 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.072907 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed2434-1dfa-46d3-908f-b9a243db93a2-metrics-certs\") pod \"network-metrics-daemon-flp4t\" (UID: \"8bed2434-1dfa-46d3-908f-b9a243db93a2\") " pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:44.073657 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.073634 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 16:17:44.083933 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.083905 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wm4\" (UniqueName: \"kubernetes.io/projected/2e0d546f-57ef-450e-920e-e707ea8876e4-kube-api-access-98wm4\") pod \"network-check-target-cf9rh\" (UID: \"2e0d546f-57ef-450e-920e-e707ea8876e4\") " pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:44.217005 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.216932 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wsr6k\"" Jul 15 16:17:44.224569 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.224542 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flp4t" Jul 15 16:17:44.328195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.328163 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dmv8w\"" Jul 15 16:17:44.335819 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.335795 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:44.876815 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.876787 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74"] Jul 15 16:17:44.880327 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.880302 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:44.883007 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.882981 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f9n8k\"" Jul 15 16:17:44.883139 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.883057 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 16:17:44.888301 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.888255 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74ff49cf57-t8fcj"] Jul 15 16:17:44.893731 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.893710 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.897266 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.897244 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 16:17:44.897393 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.897368 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kvprs\"" Jul 15 16:17:44.897651 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.897634 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 16:17:44.897764 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.897731 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 16:17:44.897764 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.897743 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 16:17:44.898595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.898579 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-16s1svetnerf0\"" Jul 15 16:17:44.900272 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.900170 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74"] Jul 15 16:17:44.909667 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.909646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74ff49cf57-t8fcj"] Jul 15 16:17:44.967215 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-secret-metrics-server-tls\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967244 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7778ad-0e65-487d-af6d-35271493696c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967272 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4a7778ad-0e65-487d-af6d-35271493696c-audit-log\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967300 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c12a20-7f52-4f3b-93ba-e71167040c45-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-nwd74\" (UID: \"80c12a20-7f52-4f3b-93ba-e71167040c45\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2z6\" (UniqueName: \"kubernetes.io/projected/4a7778ad-0e65-487d-af6d-35271493696c-kube-api-access-hp2z6\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967452 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-secret-metrics-server-client-certs\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4a7778ad-0e65-487d-af6d-35271493696c-metrics-server-audit-profiles\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967502 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l42w\" (UniqueName: \"kubernetes.io/projected/80c12a20-7f52-4f3b-93ba-e71167040c45-kube-api-access-4l42w\") pod \"monitoring-plugin-5f68df7f58-nwd74\" (UID: \"80c12a20-7f52-4f3b-93ba-e71167040c45\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:44.967600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:44.967563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-client-ca-bundle\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068657 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068628 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-secret-metrics-server-tls\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068669 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7778ad-0e65-487d-af6d-35271493696c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068686 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4a7778ad-0e65-487d-af6d-35271493696c-audit-log\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068733 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c12a20-7f52-4f3b-93ba-e71167040c45-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-nwd74\" (UID: \"80c12a20-7f52-4f3b-93ba-e71167040c45\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068797 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2z6\" (UniqueName: \"kubernetes.io/projected/4a7778ad-0e65-487d-af6d-35271493696c-kube-api-access-hp2z6\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068815 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-secret-metrics-server-client-certs\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068831 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4a7778ad-0e65-487d-af6d-35271493696c-metrics-server-audit-profiles\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.068891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068862 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l42w\" (UniqueName: \"kubernetes.io/projected/80c12a20-7f52-4f3b-93ba-e71167040c45-kube-api-access-4l42w\") pod \"monitoring-plugin-5f68df7f58-nwd74\" (UID: \"80c12a20-7f52-4f3b-93ba-e71167040c45\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:45.069289 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.068901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-client-ca-bundle\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.069289 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.069027 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4a7778ad-0e65-487d-af6d-35271493696c-audit-log\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.069687 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.069666 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a7778ad-0e65-487d-af6d-35271493696c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.069901 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.069864 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4a7778ad-0e65-487d-af6d-35271493696c-metrics-server-audit-profiles\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.071473 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.071447 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c12a20-7f52-4f3b-93ba-e71167040c45-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-nwd74\" (UID: \"80c12a20-7f52-4f3b-93ba-e71167040c45\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:45.071557 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.071492 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-client-ca-bundle\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.071632 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.071614 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-secret-metrics-server-tls\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.071805 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.071784 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4a7778ad-0e65-487d-af6d-35271493696c-secret-metrics-server-client-certs\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.082665 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.082642 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2z6\" (UniqueName: \"kubernetes.io/projected/4a7778ad-0e65-487d-af6d-35271493696c-kube-api-access-hp2z6\") pod \"metrics-server-74ff49cf57-t8fcj\" (UID: \"4a7778ad-0e65-487d-af6d-35271493696c\") " pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.082785 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.082716 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l42w\" (UniqueName: \"kubernetes.io/projected/80c12a20-7f52-4f3b-93ba-e71167040c45-kube-api-access-4l42w\") pod \"monitoring-plugin-5f68df7f58-nwd74\" (UID: \"80c12a20-7f52-4f3b-93ba-e71167040c45\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:45.192735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.192207 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:45.207808 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.206828 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:17:45.324384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.324353 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ks4kq"] Jul 15 16:17:45.350070 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.349760 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flp4t"] Jul 15 16:17:45.354522 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:45.354431 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bed2434_1dfa_46d3_908f_b9a243db93a2.slice/crio-de8d358ab14eea5dc54180df04ea92077ec0847aeb57de8d4afa889e44d6f4fb WatchSource:0}: Error finding container de8d358ab14eea5dc54180df04ea92077ec0847aeb57de8d4afa889e44d6f4fb: Status 404 returned error can't find the container with id de8d358ab14eea5dc54180df04ea92077ec0847aeb57de8d4afa889e44d6f4fb Jul 15 16:17:45.358756 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.358336 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cf9rh"] Jul 15 16:17:45.361599 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:45.361537 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0d546f_57ef_450e_920e_e707ea8876e4.slice/crio-02a2aded9a09b43323648031163682048aea5f81666cb83722cbd4f89f071355 WatchSource:0}: Error finding container 02a2aded9a09b43323648031163682048aea5f81666cb83722cbd4f89f071355: Status 404 returned error can't find the container with id 02a2aded9a09b43323648031163682048aea5f81666cb83722cbd4f89f071355 Jul 15 16:17:45.418939 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.418913 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74ff49cf57-t8fcj"] Jul 15 16:17:45.420384 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:45.420358 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7778ad_0e65_487d_af6d_35271493696c.slice/crio-a4f8d4d4ffc886adae97b9b251d524ed198495aaf96935893a3994a6228e68a6 WatchSource:0}: Error finding container a4f8d4d4ffc886adae97b9b251d524ed198495aaf96935893a3994a6228e68a6: Status 404 returned error can't find the container with id a4f8d4d4ffc886adae97b9b251d524ed198495aaf96935893a3994a6228e68a6 Jul 15 16:17:45.432063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.432041 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74"] Jul 15 16:17:45.435070 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:45.435045 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c12a20_7f52_4f3b_93ba_e71167040c45.slice/crio-5c2d7294bb8c3a89c2ba775915ab86ea0d7d00759ce267c2127d8001115fbc34 WatchSource:0}: Error finding container 5c2d7294bb8c3a89c2ba775915ab86ea0d7d00759ce267c2127d8001115fbc34: Status 404 returned error can't find the container with id 5c2d7294bb8c3a89c2ba775915ab86ea0d7d00759ce267c2127d8001115fbc34 Jul 15 16:17:45.643938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.643884 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rttt" Jul 15 16:17:45.691272 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.691167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ks4kq" event={"ID":"7b77867d-4145-4515-945f-7a50589dd046","Type":"ContainerStarted","Data":"90cc6ca893a753fe8e0053e4cd1a86eb57ea293d31d4b5fae4f9aac992d329c6"} Jul 15 16:17:45.692371 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.692337 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flp4t" event={"ID":"8bed2434-1dfa-46d3-908f-b9a243db93a2","Type":"ContainerStarted","Data":"de8d358ab14eea5dc54180df04ea92077ec0847aeb57de8d4afa889e44d6f4fb"} Jul 15 16:17:45.693660 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.693633 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" event={"ID":"4a7778ad-0e65-487d-af6d-35271493696c","Type":"ContainerStarted","Data":"a4f8d4d4ffc886adae97b9b251d524ed198495aaf96935893a3994a6228e68a6"} Jul 15 16:17:45.695632 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.695606 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1" exitCode=0 Jul 15 16:17:45.695756 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.695682 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1"} Jul 15 16:17:45.698611 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.698588 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4fdz" event={"ID":"96c0b345-1601-439b-81ba-940e5e8712b4","Type":"ContainerStarted","Data":"84485324709cef9a03fcc7afc396afd207cf75649ffcff69d4771d5879f4d2c9"} Jul 15 16:17:45.700355 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.700332 2560 generic.go:356] "Generic (PLEG): container finished" podID="99bfa9e6-40d0-4685-925f-49493b61683b" containerID="c184d69a87c45ada04906bd934282efb59513fa26435a8b6154e8cabf53e80c8" exitCode=0 Jul 15 16:17:45.700470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.700365 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjblt" event={"ID":"99bfa9e6-40d0-4685-925f-49493b61683b","Type":"ContainerDied","Data":"c184d69a87c45ada04906bd934282efb59513fa26435a8b6154e8cabf53e80c8"} Jul 15 16:17:45.702002 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.701977 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" event={"ID":"80c12a20-7f52-4f3b-93ba-e71167040c45","Type":"ContainerStarted","Data":"5c2d7294bb8c3a89c2ba775915ab86ea0d7d00759ce267c2127d8001115fbc34"} Jul 15 16:17:45.703220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.703197 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cf9rh" event={"ID":"2e0d546f-57ef-450e-920e-e707ea8876e4","Type":"ContainerStarted","Data":"02a2aded9a09b43323648031163682048aea5f81666cb83722cbd4f89f071355"} Jul 15 16:17:45.745017 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:45.744961 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x4fdz" podStartSLOduration=1.411281933 podStartE2EDuration="17.744940396s" totalImagesPullingTime="16.333658463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:17:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:45.744348875 +0000 UTC m=+67.999249085" watchObservedRunningTime="2026-07-15 16:17:45.744940396 +0000 UTC m=+67.999840607" Jul 15 16:17:46.718059 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:46.715700 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjblt" event={"ID":"99bfa9e6-40d0-4685-925f-49493b61683b","Type":"ContainerStarted","Data":"f254bd4fd7a2413acf66880dbdc944edb396fa99808fb4535130cdedfd370ffc"} Jul 15 16:17:46.718059 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:46.715762 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjblt" event={"ID":"99bfa9e6-40d0-4685-925f-49493b61683b","Type":"ContainerStarted","Data":"f5a46c98b7eada1f8aaa3aca955c40dd09335b8bd41eca52f38a96984de9484b"} Jul 15 16:17:46.746230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:46.745116 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hjblt" podStartSLOduration=3.6498761220000002 podStartE2EDuration="6.745096798s" totalImagesPullingTime="2.971527357s" totalInitContainerRuntime="123.693319ms" isStatefulPod=true podCreationTimestamp="2026-07-15 16:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:46.744216993 +0000 UTC m=+68.999117200" watchObservedRunningTime="2026-07-15 16:17:46.745096798 +0000 UTC m=+68.999997006" Jul 15 16:17:48.668429 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.668191 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-684866f46d-bmnds"] Jul 15 16:17:48.671522 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.671448 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.684373 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.684351 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 16:17:48.685838 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.685810 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 16:17:48.685838 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.685825 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 16:17:48.686032 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.685966 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 16:17:48.686137 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.686120 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 16:17:48.686197 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.686148 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 16:17:48.686197 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.686179 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vpr9m\"" Jul 15 16:17:48.686342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.686179 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 16:17:48.706309 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.706255 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684866f46d-bmnds"] Jul 15 16:17:48.806851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.806818 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-serving-cert\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.807013 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.806857 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-config\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.807013 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.806914 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-oauth-config\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.807013 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.806954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-service-ca\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.807013 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.806978 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5vbh\" (UniqueName: \"kubernetes.io/projected/92b720f1-5a25-4953-bf5a-ac07064f9f56-kube-api-access-s5vbh\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.807171 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.807040 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-oauth-serving-cert\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.908169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.908135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-serving-cert\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.908169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.908177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-config\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.908399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.908213 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-oauth-config\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.908399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.908253 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-service-ca\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.908399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.908277 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5vbh\" (UniqueName: \"kubernetes.io/projected/92b720f1-5a25-4953-bf5a-ac07064f9f56-kube-api-access-s5vbh\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.908399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.908317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-oauth-serving-cert\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.909470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.909115 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-oauth-serving-cert\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.909470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.909207 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-service-ca\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.910230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.910143 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-config\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.912408 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.912227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-oauth-config\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.912408 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.912378 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-serving-cert\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.922496 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.922444 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5vbh\" (UniqueName: \"kubernetes.io/projected/92b720f1-5a25-4953-bf5a-ac07064f9f56-kube-api-access-s5vbh\") pod \"console-684866f46d-bmnds\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:48.986336 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:48.986299 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:51.659974 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.659798 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684866f46d-bmnds"] Jul 15 16:17:51.745036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.743883 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cf9rh" event={"ID":"2e0d546f-57ef-450e-920e-e707ea8876e4","Type":"ContainerStarted","Data":"08794fe3f3c23898247aa10df0f14d7475b479237fbd8cd495fd8b699fda02ed"} Jul 15 16:17:51.745036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.744145 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:17:51.750665 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.750198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ks4kq" event={"ID":"7b77867d-4145-4515-945f-7a50589dd046","Type":"ContainerStarted","Data":"9e7c056daf00cab57fa9bf7fd7fbfd2d9cc0e971fa4a4272db6ac95cbc959b40"} Jul 15 16:17:51.754622 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.754528 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flp4t" event={"ID":"8bed2434-1dfa-46d3-908f-b9a243db93a2","Type":"ContainerStarted","Data":"7930b3bfcd21441a6f21689604e56dd534ceb86dee7ccb08e56639a8caab0251"} Jul 15 16:17:51.757509 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.757385 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"805cfc0795cc755f5a33e453e93691ccd249a45a0fbb630a9effca0afc057507"} Jul 15 16:17:51.757509 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.757446 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"4931fd2f2149427bf84ff9c74a761bd0c2214767518e0b2ac5048925d415fb6f"} Jul 15 16:17:51.760160 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.759660 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" event={"ID":"4a7778ad-0e65-487d-af6d-35271493696c","Type":"ContainerStarted","Data":"043bd0eda759f67c3811299dc2ed0bf7a0cf0b9a1001d21a39453ea9f664e9c5"} Jul 15 16:17:51.762610 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.762594 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684866f46d-bmnds" event={"ID":"92b720f1-5a25-4953-bf5a-ac07064f9f56","Type":"ContainerStarted","Data":"4b615c5cb3f4a5eaf262ec24a0431d80d07a9d68060cd670ce535555061c856e"} Jul 15 16:17:51.765603 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.765586 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637"} Jul 15 16:17:51.780293 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.773650 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" event={"ID":"80c12a20-7f52-4f3b-93ba-e71167040c45","Type":"ContainerStarted","Data":"2480b03bc227681acd0a3e99a00aaaa17220213f7587355ddebdb08b1a7b7fb2"} Jul 15 16:17:51.780293 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.775725 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:51.788619 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.787906 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" Jul 15 16:17:51.793124 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.792702 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cf9rh" podStartSLOduration=67.672405486 podStartE2EDuration="1m13.792683094s" totalImagesPullingTime="6.120277608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:51.76948074 +0000 UTC m=+74.024380950" watchObservedRunningTime="2026-07-15 16:17:51.792683094 +0000 UTC m=+74.047583285" Jul 15 16:17:51.794175 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.793853 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" podStartSLOduration=1.7323846120000002 podStartE2EDuration="7.793840797s" totalImagesPullingTime="6.061456185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:51.791742241 +0000 UTC m=+74.046642447" watchObservedRunningTime="2026-07-15 16:17:51.793840797 +0000 UTC m=+74.048741005" Jul 15 16:17:51.812724 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:51.812672 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ks4kq" podStartSLOduration=67.640311884 podStartE2EDuration="1m13.812652123s" totalImagesPullingTime="6.172340239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:51.812047743 +0000 UTC m=+74.066947950" watchObservedRunningTime="2026-07-15 16:17:51.812652123 +0000 UTC m=+74.067552330" Jul 15 16:17:52.778518 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.778479 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flp4t" event={"ID":"8bed2434-1dfa-46d3-908f-b9a243db93a2","Type":"ContainerStarted","Data":"1b684b91955d425fd971772b22f2e7d0a88dee9c6b6fefc7defa20e912312846"} Jul 15 16:17:52.780533 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.780510 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"d46c72f6373e7d067499e14124ebf18c093f88ee9bbae478390d060b52b5fd0f"} Jul 15 16:17:52.783063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.783031 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7"} Jul 15 16:17:52.783171 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.783095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4"} Jul 15 16:17:52.783171 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.783124 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f"} Jul 15 16:17:52.783171 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.783147 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa"} Jul 15 16:17:52.798172 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.798130 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-flp4t" podStartSLOduration=68.682156547 podStartE2EDuration="1m14.798118246s" totalImagesPullingTime="6.115961699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:52.797068697 +0000 UTC m=+75.051968924" watchObservedRunningTime="2026-07-15 16:17:52.798118246 +0000 UTC m=+75.053018452" Jul 15 16:17:52.798779 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:52.798752 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nwd74" podStartSLOduration=2.752212567 podStartE2EDuration="8.798740737s" totalImagesPullingTime="6.04652817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:51.83148586 +0000 UTC m=+74.086386067" watchObservedRunningTime="2026-07-15 16:17:52.798740737 +0000 UTC m=+75.053640943" Jul 15 16:17:54.796683 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.796637 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"3560caf28d23407cfb527489765a2714beb3616d215a803092062a4b2deab28a"} Jul 15 16:17:54.797166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.796723 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"6dcab20cac55ce62cd4700999cac4e828e8d147cbcbb0033104809ae5432f173"} Jul 15 16:17:54.797166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.796768 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" event={"ID":"4eca1df6-3d6d-480b-bbee-abfb87e4bece","Type":"ContainerStarted","Data":"9e704de79d3c374d5f5e060706fde4f1f5424dfd36405d2e90e44a13e60ca392"} Jul 15 16:17:54.797166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.797034 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:17:54.800453 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.800428 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerStarted","Data":"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109"} Jul 15 16:17:54.823946 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.823900 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" podStartSLOduration=2.164305083 podStartE2EDuration="12.823885103s" totalImagesPullingTime="10.65958002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:54.821210617 +0000 UTC m=+77.076110877" watchObservedRunningTime="2026-07-15 16:17:54.823885103 +0000 UTC m=+77.078785302" Jul 15 16:17:54.851273 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:54.851223 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.815018686 podStartE2EDuration="13.851207464s" totalImagesPullingTime="11.006960293s" totalInitContainerRuntime="29.228485ms" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:54.849261008 +0000 UTC m=+77.104161214" watchObservedRunningTime="2026-07-15 16:17:54.851207464 +0000 UTC m=+77.106107670" Jul 15 16:17:55.805789 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.805749 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684866f46d-bmnds" event={"ID":"92b720f1-5a25-4953-bf5a-ac07064f9f56","Type":"ContainerStarted","Data":"1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541"} Jul 15 16:17:55.848234 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.848197 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5bf4994-ls6b9"] Jul 15 16:17:55.849071 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.849032 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-684866f46d-bmnds" podStartSLOduration=4.392855196 podStartE2EDuration="7.849018869s" totalImagesPullingTime="3.456163673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:55.841505804 +0000 UTC m=+78.096406010" watchObservedRunningTime="2026-07-15 16:17:55.849018869 +0000 UTC m=+78.103919075" Jul 15 16:17:55.851408 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.851391 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.866175 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.866154 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 16:17:55.870561 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.870538 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5bf4994-ls6b9"] Jul 15 16:17:55.969703 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.969679 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-oauth-serving-cert\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.970284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.970078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-trusted-ca-bundle\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.970284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.970150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-service-ca\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.970284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.970230 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-oauth-config\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.970510 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.970464 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkn9\" (UniqueName: \"kubernetes.io/projected/6826515f-c986-431b-b272-4fda2e45e2b1-kube-api-access-8pkn9\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.970661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.970645 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-serving-cert\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:55.970785 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:55.970763 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-console-config\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-console-config\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071435 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-oauth-serving-cert\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-trusted-ca-bundle\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071645 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-service-ca\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071645 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071498 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-oauth-config\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071645 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkn9\" (UniqueName: \"kubernetes.io/projected/6826515f-c986-431b-b272-4fda2e45e2b1-kube-api-access-8pkn9\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.071645 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.071570 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-serving-cert\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.072273 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.072243 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-oauth-serving-cert\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.072273 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.072256 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-console-config\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.072273 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.072266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-service-ca\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.072481 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.072447 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-trusted-ca-bundle\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.073946 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.073923 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-oauth-config\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.074104 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.074090 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-serving-cert\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.083414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.083394 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkn9\" (UniqueName: \"kubernetes.io/projected/6826515f-c986-431b-b272-4fda2e45e2b1-kube-api-access-8pkn9\") pod \"console-6f5bf4994-ls6b9\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.160555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.160524 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:17:56.293775 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.293754 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5bf4994-ls6b9"] Jul 15 16:17:56.296250 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:17:56.296215 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6826515f_c986_431b_b272_4fda2e45e2b1.slice/crio-3f8d48eeef241af3cf7f01649e26c3b7998f0db062cec527e82b7a14377a66e5 WatchSource:0}: Error finding container 3f8d48eeef241af3cf7f01649e26c3b7998f0db062cec527e82b7a14377a66e5: Status 404 returned error can't find the container with id 3f8d48eeef241af3cf7f01649e26c3b7998f0db062cec527e82b7a14377a66e5 Jul 15 16:17:56.811446 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.811413 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5bf4994-ls6b9" event={"ID":"6826515f-c986-431b-b272-4fda2e45e2b1","Type":"ContainerStarted","Data":"4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f"} Jul 15 16:17:56.811911 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.811469 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5bf4994-ls6b9" event={"ID":"6826515f-c986-431b-b272-4fda2e45e2b1","Type":"ContainerStarted","Data":"3f8d48eeef241af3cf7f01649e26c3b7998f0db062cec527e82b7a14377a66e5"} Jul 15 16:17:56.832440 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:56.832400 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5bf4994-ls6b9" podStartSLOduration=1.832386378 podStartE2EDuration="1.832386378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:17:56.831107498 +0000 UTC m=+79.086007709" watchObservedRunningTime="2026-07-15 16:17:56.832386378 +0000 UTC m=+79.087286583" Jul 15 16:17:58.987524 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:58.987488 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:58.987989 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:58.987556 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:58.992452 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:58.992429 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:17:59.829807 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:17:59.829783 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:18:00.811650 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:00.811625 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d8f5586cc-gfpkk" Jul 15 16:18:05.207311 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:05.207270 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:18:05.207760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:05.207339 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:18:06.160917 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:06.160883 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:18:06.161150 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:06.160936 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:18:06.165577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:06.165555 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:18:06.858539 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:06.858512 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:18:06.908690 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:06.908661 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684866f46d-bmnds"] Jul 15 16:18:22.786328 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:22.786299 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cf9rh" Jul 15 16:18:25.213353 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:25.213324 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:18:25.218416 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:25.218393 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74ff49cf57-t8fcj" Jul 15 16:18:31.930981 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:31.930938 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-684866f46d-bmnds" podUID="92b720f1-5a25-4953-bf5a-ac07064f9f56" containerName="console" containerID="cri-o://1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541" gracePeriod=15 Jul 15 16:18:32.163811 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.163788 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:18:32.251378 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251318 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-serving-cert\") pod \"92b720f1-5a25-4953-bf5a-ac07064f9f56\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " Jul 15 16:18:32.251378 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251352 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5vbh\" (UniqueName: \"kubernetes.io/projected/92b720f1-5a25-4953-bf5a-ac07064f9f56-kube-api-access-s5vbh\") pod \"92b720f1-5a25-4953-bf5a-ac07064f9f56\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " Jul 15 16:18:32.251378 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251374 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-oauth-serving-cert\") pod \"92b720f1-5a25-4953-bf5a-ac07064f9f56\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " Jul 15 16:18:32.251664 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251405 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-config\") pod \"92b720f1-5a25-4953-bf5a-ac07064f9f56\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " Jul 15 16:18:32.251664 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251461 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-service-ca\") pod \"92b720f1-5a25-4953-bf5a-ac07064f9f56\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " Jul 15 16:18:32.251664 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251498 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-oauth-config\") pod \"92b720f1-5a25-4953-bf5a-ac07064f9f56\" (UID: \"92b720f1-5a25-4953-bf5a-ac07064f9f56\") " Jul 15 16:18:32.251823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251782 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "92b720f1-5a25-4953-bf5a-ac07064f9f56" (UID: "92b720f1-5a25-4953-bf5a-ac07064f9f56"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:18:32.251904 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251830 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-config" (OuterVolumeSpecName: "console-config") pod "92b720f1-5a25-4953-bf5a-ac07064f9f56" (UID: "92b720f1-5a25-4953-bf5a-ac07064f9f56"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:18:32.251904 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.251843 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-service-ca" (OuterVolumeSpecName: "service-ca") pod "92b720f1-5a25-4953-bf5a-ac07064f9f56" (UID: "92b720f1-5a25-4953-bf5a-ac07064f9f56"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:18:32.253669 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.253645 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "92b720f1-5a25-4953-bf5a-ac07064f9f56" (UID: "92b720f1-5a25-4953-bf5a-ac07064f9f56"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:18:32.253947 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.253928 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "92b720f1-5a25-4953-bf5a-ac07064f9f56" (UID: "92b720f1-5a25-4953-bf5a-ac07064f9f56"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:18:32.254007 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.253978 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b720f1-5a25-4953-bf5a-ac07064f9f56-kube-api-access-s5vbh" (OuterVolumeSpecName: "kube-api-access-s5vbh") pod "92b720f1-5a25-4953-bf5a-ac07064f9f56" (UID: "92b720f1-5a25-4953-bf5a-ac07064f9f56"). InnerVolumeSpecName "kube-api-access-s5vbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:18:32.352136 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.352112 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:18:32.352226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.352149 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5vbh\" (UniqueName: \"kubernetes.io/projected/92b720f1-5a25-4953-bf5a-ac07064f9f56-kube-api-access-s5vbh\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:18:32.352226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.352159 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-oauth-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:18:32.352226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.352168 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:18:32.352226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.352177 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92b720f1-5a25-4953-bf5a-ac07064f9f56-service-ca\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:18:32.352226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.352184 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92b720f1-5a25-4953-bf5a-ac07064f9f56-console-oauth-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:18:32.954696 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.954662 2560 generic.go:356] "Generic (PLEG): container finished" podID="92b720f1-5a25-4953-bf5a-ac07064f9f56" containerID="1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541" exitCode=0 Jul 15 16:18:32.955168 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.954710 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684866f46d-bmnds" event={"ID":"92b720f1-5a25-4953-bf5a-ac07064f9f56","Type":"ContainerDied","Data":"1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541"} Jul 15 16:18:32.955168 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.954718 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684866f46d-bmnds" Jul 15 16:18:32.955168 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.954787 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684866f46d-bmnds" event={"ID":"92b720f1-5a25-4953-bf5a-ac07064f9f56","Type":"ContainerDied","Data":"4b615c5cb3f4a5eaf262ec24a0431d80d07a9d68060cd670ce535555061c856e"} Jul 15 16:18:32.955168 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.954809 2560 scope.go:117] "RemoveContainer" containerID="1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541" Jul 15 16:18:32.962971 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.962913 2560 scope.go:117] "RemoveContainer" containerID="1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541" Jul 15 16:18:32.963232 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:18:32.963206 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541\": container with ID starting with 1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541 not found: ID does not exist" containerID="1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541" Jul 15 16:18:32.963356 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.963248 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541"} err="failed to get container status \"1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541\": rpc error: code = NotFound desc = could not find container \"1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541\": container with ID starting with 1e67131a40feac53486d63662312297614d64592a8b3462e10cedf241caaf541 not found: ID does not exist" Jul 15 16:18:32.979143 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.979122 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684866f46d-bmnds"] Jul 15 16:18:32.984620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:32.984601 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-684866f46d-bmnds"] Jul 15 16:18:34.309424 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:18:34.309391 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b720f1-5a25-4953-bf5a-ac07064f9f56" path="/var/lib/kubelet/pods/92b720f1-5a25-4953-bf5a-ac07064f9f56/volumes" Jul 15 16:19:00.591744 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.591706 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:19:00.592181 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.592015 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="alertmanager" containerID="cri-o://2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637" gracePeriod=120 Jul 15 16:19:00.592181 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.592097 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-web" containerID="cri-o://a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f" gracePeriod=120 Jul 15 16:19:00.592181 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.592122 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy" containerID="cri-o://07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4" gracePeriod=120 Jul 15 16:19:00.592181 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.592159 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="prom-label-proxy" containerID="cri-o://c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109" gracePeriod=120 Jul 15 16:19:00.592404 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.592150 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="config-reloader" containerID="cri-o://af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa" gracePeriod=120 Jul 15 16:19:00.592404 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:00.592098 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-metric" containerID="cri-o://ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7" gracePeriod=120 Jul 15 16:19:01.059595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059508 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109" exitCode=0 Jul 15 16:19:01.059595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059535 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4" exitCode=0 Jul 15 16:19:01.059595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059542 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa" exitCode=0 Jul 15 16:19:01.059595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059547 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637" exitCode=0 Jul 15 16:19:01.059595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059564 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109"} Jul 15 16:19:01.059845 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059616 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4"} Jul 15 16:19:01.059845 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059642 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa"} Jul 15 16:19:01.059845 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.059664 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637"} Jul 15 16:19:01.839019 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.838996 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:01.979359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979281 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-main-db\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979323 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979355 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-config-out\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979418 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979464 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-trusted-ca-bundle\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979488 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-web-config\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979538 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcm4f\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-kube-api-access-dcm4f\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979567 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-config-volume\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979596 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-cluster-tls-config\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979626 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-web\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979639 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:19:01.979661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979651 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-main-tls\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.980123 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979677 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-metrics-client-ca\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.980123 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979713 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-tls-assets\") pod \"3709ae13-90a7-4f38-a159-5e55fadc963b\" (UID: \"3709ae13-90a7-4f38-a159-5e55fadc963b\") " Jul 15 16:19:01.980123 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979908 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:19:01.980123 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.979926 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-main-db\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:01.982505 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.982331 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-config-out" (OuterVolumeSpecName: "config-out") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:19:01.983069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.982663 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:19:01.983069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.982707 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-config-volume" (OuterVolumeSpecName: "config-volume") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:01.983069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.982756 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:01.983069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.982828 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:01.983069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.983030 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:01.983069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.983056 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:19:01.983333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.983140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:01.984912 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.984887 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-kube-api-access-dcm4f" (OuterVolumeSpecName: "kube-api-access-dcm4f") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "kube-api-access-dcm4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:19:01.988075 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.988051 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:01.993235 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:01.993212 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-web-config" (OuterVolumeSpecName: "web-config") pod "3709ae13-90a7-4f38-a159-5e55fadc963b" (UID: "3709ae13-90a7-4f38-a159-5e55fadc963b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:02.066419 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066389 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7" exitCode=0 Jul 15 16:19:02.066419 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066411 2560 generic.go:356] "Generic (PLEG): container finished" podID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerID="a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f" exitCode=0 Jul 15 16:19:02.066580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066474 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.066580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066479 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7"} Jul 15 16:19:02.066580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066549 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f"} Jul 15 16:19:02.066580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066577 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3709ae13-90a7-4f38-a159-5e55fadc963b","Type":"ContainerDied","Data":"48f49d3ac972ae7190563802f655acad4f823e2d6093f80a8f91727cb2bc48d6"} Jul 15 16:19:02.066703 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.066609 2560 scope.go:117] "RemoveContainer" containerID="c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109" Jul 15 16:19:02.075070 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.075048 2560 scope.go:117] "RemoveContainer" containerID="ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7" Jul 15 16:19:02.080263 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080240 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080263 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080261 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-web-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080271 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcm4f\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-kube-api-access-dcm4f\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080279 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-config-volume\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080287 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-cluster-tls-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080296 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080305 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-main-tls\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080313 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3709ae13-90a7-4f38-a159-5e55fadc963b-metrics-client-ca\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080321 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3709ae13-90a7-4f38-a159-5e55fadc963b-tls-assets\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080329 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080337 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3709ae13-90a7-4f38-a159-5e55fadc963b-config-out\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.080417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.080346 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3709ae13-90a7-4f38-a159-5e55fadc963b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:02.082034 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.082020 2560 scope.go:117] "RemoveContainer" containerID="07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4" Jul 15 16:19:02.088560 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.088538 2560 scope.go:117] "RemoveContainer" containerID="a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f" Jul 15 16:19:02.090948 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.090928 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:19:02.097316 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.097298 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:19:02.110566 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.110552 2560 scope.go:117] "RemoveContainer" containerID="af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa" Jul 15 16:19:02.117603 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.117580 2560 scope.go:117] "RemoveContainer" containerID="2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637" Jul 15 16:19:02.123884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.123851 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:19:02.124240 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124225 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-metric" Jul 15 16:19:02.124294 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124244 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-metric" Jul 15 16:19:02.124294 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124256 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b720f1-5a25-4953-bf5a-ac07064f9f56" containerName="console" Jul 15 16:19:02.124294 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124266 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b720f1-5a25-4953-bf5a-ac07064f9f56" containerName="console" Jul 15 16:19:02.124294 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124286 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124295 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124307 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="config-reloader" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124316 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="config-reloader" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124326 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="prom-label-proxy" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124334 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="prom-label-proxy" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124349 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-web" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124357 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-web" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124371 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="alertmanager" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124378 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="alertmanager" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124388 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="init-config-reloader" Jul 15 16:19:02.124431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124396 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="init-config-reloader" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124467 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="config-reloader" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124482 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="alertmanager" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124491 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-web" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124500 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy-metric" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124509 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="kube-rbac-proxy" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124518 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" containerName="prom-label-proxy" Jul 15 16:19:02.124769 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.124525 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b720f1-5a25-4953-bf5a-ac07064f9f56" containerName="console" Jul 15 16:19:02.125800 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.125779 2560 scope.go:117] "RemoveContainer" containerID="38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1" Jul 15 16:19:02.128761 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.128743 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.131946 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.131971 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.132004 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.132013 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.132044 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9m695\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.131949 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.131950 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.132103 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 16:19:02.132167 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.131946 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 16:19:02.133842 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.133824 2560 scope.go:117] "RemoveContainer" containerID="c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109" Jul 15 16:19:02.134149 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.134127 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109\": container with ID starting with c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109 not found: ID does not exist" containerID="c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109" Jul 15 16:19:02.134230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.134160 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109"} err="failed to get container status \"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109\": rpc error: code = NotFound desc = could not find container \"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109\": container with ID starting with c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109 not found: ID does not exist" Jul 15 16:19:02.134230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.134177 2560 scope.go:117] "RemoveContainer" containerID="ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7" Jul 15 16:19:02.134457 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.134440 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7\": container with ID starting with ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7 not found: ID does not exist" containerID="ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7" Jul 15 16:19:02.134501 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.134460 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7"} err="failed to get container status \"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7\": rpc error: code = NotFound desc = could not find container \"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7\": container with ID starting with ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7 not found: ID does not exist" Jul 15 16:19:02.134501 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.134473 2560 scope.go:117] "RemoveContainer" containerID="07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4" Jul 15 16:19:02.134787 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.134750 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4\": container with ID starting with 07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4 not found: ID does not exist" containerID="07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4" Jul 15 16:19:02.134884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.134778 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4"} err="failed to get container status \"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4\": rpc error: code = NotFound desc = could not find container \"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4\": container with ID starting with 07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4 not found: ID does not exist" Jul 15 16:19:02.134884 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.134800 2560 scope.go:117] "RemoveContainer" containerID="a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f" Jul 15 16:19:02.135521 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.135471 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f\": container with ID starting with a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f not found: ID does not exist" containerID="a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f" Jul 15 16:19:02.135600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.135528 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f"} err="failed to get container status \"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f\": rpc error: code = NotFound desc = could not find container \"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f\": container with ID starting with a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f not found: ID does not exist" Jul 15 16:19:02.135600 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.135566 2560 scope.go:117] "RemoveContainer" containerID="af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa" Jul 15 16:19:02.135909 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.135860 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa\": container with ID starting with af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa not found: ID does not exist" containerID="af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa" Jul 15 16:19:02.136014 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.135915 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa"} err="failed to get container status \"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa\": rpc error: code = NotFound desc = could not find container \"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa\": container with ID starting with af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa not found: ID does not exist" Jul 15 16:19:02.136014 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.135937 2560 scope.go:117] "RemoveContainer" containerID="2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637" Jul 15 16:19:02.136249 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.136202 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637\": container with ID starting with 2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637 not found: ID does not exist" containerID="2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637" Jul 15 16:19:02.136249 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.136228 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637"} err="failed to get container status \"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637\": rpc error: code = NotFound desc = could not find container \"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637\": container with ID starting with 2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637 not found: ID does not exist" Jul 15 16:19:02.136249 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.136247 2560 scope.go:117] "RemoveContainer" containerID="38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1" Jul 15 16:19:02.136576 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:02.136550 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1\": container with ID starting with 38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1 not found: ID does not exist" containerID="38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1" Jul 15 16:19:02.136665 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.136595 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1"} err="failed to get container status \"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1\": rpc error: code = NotFound desc = could not find container \"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1\": container with ID starting with 38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1 not found: ID does not exist" Jul 15 16:19:02.136665 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.136617 2560 scope.go:117] "RemoveContainer" containerID="c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109" Jul 15 16:19:02.136922 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.136904 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109"} err="failed to get container status \"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109\": rpc error: code = NotFound desc = could not find container \"c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109\": container with ID starting with c9250eb876cee953aff4f834bb7b45b887674e8bb0179b94840c6b8d41aad109 not found: ID does not exist" Jul 15 16:19:02.136985 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.136922 2560 scope.go:117] "RemoveContainer" containerID="ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7" Jul 15 16:19:02.137180 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137160 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7"} err="failed to get container status \"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7\": rpc error: code = NotFound desc = could not find container \"ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7\": container with ID starting with ce5dd9f6f77e1cfc404b17a13d9252fd2c6e359b6169fbbfb4951a85ceffc3b7 not found: ID does not exist" Jul 15 16:19:02.137256 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137182 2560 scope.go:117] "RemoveContainer" containerID="07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4" Jul 15 16:19:02.137418 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137401 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4"} err="failed to get container status \"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4\": rpc error: code = NotFound desc = could not find container \"07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4\": container with ID starting with 07fa1db3784715b0621328a86dc3632fd03e7db3e3f06792587f2858669d4ec4 not found: ID does not exist" Jul 15 16:19:02.137470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137420 2560 scope.go:117] "RemoveContainer" containerID="a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f" Jul 15 16:19:02.137613 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137599 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 16:19:02.137679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137646 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f"} err="failed to get container status \"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f\": rpc error: code = NotFound desc = could not find container \"a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f\": container with ID starting with a3d5cd55388bf9b7abe03b0f932e86976405021a35b5703813c5b469d418184f not found: ID does not exist" Jul 15 16:19:02.137679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137667 2560 scope.go:117] "RemoveContainer" containerID="af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa" Jul 15 16:19:02.137950 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137929 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa"} err="failed to get container status \"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa\": rpc error: code = NotFound desc = could not find container \"af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa\": container with ID starting with af9dce35ab637ef4bbe7960b97fa4325507b4c61b78c7129b118012eb6d675fa not found: ID does not exist" Jul 15 16:19:02.138008 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.137954 2560 scope.go:117] "RemoveContainer" containerID="2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637" Jul 15 16:19:02.138185 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.138168 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637"} err="failed to get container status \"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637\": rpc error: code = NotFound desc = could not find container \"2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637\": container with ID starting with 2aa36e0a3eb8af1d1204d6666f62dfe1319f4ac50fe615fe6ce991e0c4623637 not found: ID does not exist" Jul 15 16:19:02.138231 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.138185 2560 scope.go:117] "RemoveContainer" containerID="38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1" Jul 15 16:19:02.138398 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.138381 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1"} err="failed to get container status \"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1\": rpc error: code = NotFound desc = could not find container \"38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1\": container with ID starting with 38707a3007a01ced0ccca590cfeb4e41f85978d8788b4aeab8bbc6b76275d6b1 not found: ID does not exist" Jul 15 16:19:02.150648 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.150618 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:19:02.281894 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.281784 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.281894 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.281824 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.281913 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.281960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxlt\" (UniqueName: \"kubernetes.io/projected/85b15886-8010-49c4-9fce-7ceca9102af9-kube-api-access-bzxlt\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.281997 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85b15886-8010-49c4-9fce-7ceca9102af9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282013 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85b15886-8010-49c4-9fce-7ceca9102af9-config-out\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282029 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-web-config\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282097 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85b15886-8010-49c4-9fce-7ceca9102af9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282140 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b15886-8010-49c4-9fce-7ceca9102af9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282160 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b15886-8010-49c4-9fce-7ceca9102af9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.282251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.282176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-config-volume\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.309455 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.309424 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3709ae13-90a7-4f38-a159-5e55fadc963b" path="/var/lib/kubelet/pods/3709ae13-90a7-4f38-a159-5e55fadc963b/volumes" Jul 15 16:19:02.383033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383151 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383040 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383151 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383109 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383426 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxlt\" (UniqueName: \"kubernetes.io/projected/85b15886-8010-49c4-9fce-7ceca9102af9-kube-api-access-bzxlt\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383501 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85b15886-8010-49c4-9fce-7ceca9102af9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85b15886-8010-49c4-9fce-7ceca9102af9-config-out\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-web-config\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383704 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383588 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383704 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85b15886-8010-49c4-9fce-7ceca9102af9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383704 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383655 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b15886-8010-49c4-9fce-7ceca9102af9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383852 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b15886-8010-49c4-9fce-7ceca9102af9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.383852 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.383732 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-config-volume\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.384378 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.384351 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b15886-8010-49c4-9fce-7ceca9102af9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.384807 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.384782 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/85b15886-8010-49c4-9fce-7ceca9102af9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.385862 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.385832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b15886-8010-49c4-9fce-7ceca9102af9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386206 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386173 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85b15886-8010-49c4-9fce-7ceca9102af9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386292 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386292 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386265 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386370 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386437 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-config-volume\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386821 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386801 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85b15886-8010-49c4-9fce-7ceca9102af9-config-out\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.386925 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.386903 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.387992 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.387972 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.388099 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.388085 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85b15886-8010-49c4-9fce-7ceca9102af9-web-config\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.392029 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.392014 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxlt\" (UniqueName: \"kubernetes.io/projected/85b15886-8010-49c4-9fce-7ceca9102af9-kube-api-access-bzxlt\") pod \"alertmanager-main-0\" (UID: \"85b15886-8010-49c4-9fce-7ceca9102af9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.439083 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.439041 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 16:19:02.578576 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:02.578500 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 16:19:02.581535 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:19:02.581510 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b15886_8010_49c4_9fce_7ceca9102af9.slice/crio-6f8161e5f36392bb0702bee5a9f33723822de576e0b7fc9768c8c3044cc7a122 WatchSource:0}: Error finding container 6f8161e5f36392bb0702bee5a9f33723822de576e0b7fc9768c8c3044cc7a122: Status 404 returned error can't find the container with id 6f8161e5f36392bb0702bee5a9f33723822de576e0b7fc9768c8c3044cc7a122 Jul 15 16:19:03.071526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:03.071494 2560 generic.go:356] "Generic (PLEG): container finished" podID="85b15886-8010-49c4-9fce-7ceca9102af9" containerID="23a5a84d3714efb94fb4d445a75921cbf18d4daaf174b086de74e23662a67c33" exitCode=0 Jul 15 16:19:03.072021 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:03.071578 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerDied","Data":"23a5a84d3714efb94fb4d445a75921cbf18d4daaf174b086de74e23662a67c33"} Jul 15 16:19:03.072021 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:03.071667 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"6f8161e5f36392bb0702bee5a9f33723822de576e0b7fc9768c8c3044cc7a122"} Jul 15 16:19:04.079923 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.079890 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"11112765d36ca29d452eb75319db6bbee201967ae9da18a385645fde9cabc374"} Jul 15 16:19:04.080305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.079954 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"98cc659a9620e5558f25e38cf110720d746e7bc919e637294ed386aac7282fda"} Jul 15 16:19:04.080305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.079983 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"07a98bba2d851c0788167afa04eb40c55741066b230cbdc2d0696a6a5081baea"} Jul 15 16:19:04.080305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.080008 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"f521ad614661b4cb9e28b19f0369fd16b56b5f938cfcbca808b4066c0e082460"} Jul 15 16:19:04.080305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.080036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"084aa39ea0fb08f409dcebc975d6616b3d4493ab9da9e92def1651c47771d7a8"} Jul 15 16:19:04.080305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.080065 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"85b15886-8010-49c4-9fce-7ceca9102af9","Type":"ContainerStarted","Data":"6503feacf889606a43569b2cb28ea0ab1203246eefd1151c69f914060efa1e9a"} Jul 15 16:19:04.141037 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.140988 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.125327555 podStartE2EDuration="2.140968281s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.640726ms" isStatefulPod=false podCreationTimestamp="2026-07-15 16:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:19:04.14082016 +0000 UTC m=+146.395720389" watchObservedRunningTime="2026-07-15 16:19:04.140968281 +0000 UTC m=+146.395868481" Jul 15 16:19:04.665471 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.665434 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dc877f5db-wdhq7"] Jul 15 16:19:04.668363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.668344 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.671607 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.671578 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 16:19:04.671886 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.671854 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 16:19:04.672094 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.672076 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 16:19:04.672297 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.672274 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 16:19:04.672438 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.672410 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-d6cns\"" Jul 15 16:19:04.672554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.672503 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 16:19:04.679712 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.679690 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 16:19:04.682857 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.682833 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc877f5db-wdhq7"] Jul 15 16:19:04.804572 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804535 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804572 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804575 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-serving-certs-ca-bundle\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804783 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804783 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804691 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpnj\" (UniqueName: \"kubernetes.io/projected/e39255c0-38b0-40ed-896f-bf7b212abddd-kube-api-access-qnpnj\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804783 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804739 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-secret-telemeter-client\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804783 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804764 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-federate-client-tls\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804930 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804789 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-telemeter-client-tls\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.804930 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.804808 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-metrics-client-ca\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905547 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905515 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-serving-certs-ca-bundle\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905547 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905601 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpnj\" (UniqueName: \"kubernetes.io/projected/e39255c0-38b0-40ed-896f-bf7b212abddd-kube-api-access-qnpnj\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-secret-telemeter-client\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-federate-client-tls\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-telemeter-client-tls\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-metrics-client-ca\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.905919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.905773 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.906362 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.906338 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-serving-certs-ca-bundle\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.906552 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.906523 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.906552 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.906548 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e39255c0-38b0-40ed-896f-bf7b212abddd-metrics-client-ca\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.908434 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.908410 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.908713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.908689 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-federate-client-tls\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.908800 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.908763 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-telemeter-client-tls\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.908800 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.908768 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e39255c0-38b0-40ed-896f-bf7b212abddd-secret-telemeter-client\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.915642 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.915569 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpnj\" (UniqueName: \"kubernetes.io/projected/e39255c0-38b0-40ed-896f-bf7b212abddd-kube-api-access-qnpnj\") pod \"telemeter-client-6dc877f5db-wdhq7\" (UID: \"e39255c0-38b0-40ed-896f-bf7b212abddd\") " pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:04.986413 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:04.986380 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" Jul 15 16:19:05.131886 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:05.131827 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc877f5db-wdhq7"] Jul 15 16:19:05.136517 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:19:05.136489 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39255c0_38b0_40ed_896f_bf7b212abddd.slice/crio-092bb89b928cf128aa231bc081baa685035d7a0d9495101ea1d4a55ab56ba008 WatchSource:0}: Error finding container 092bb89b928cf128aa231bc081baa685035d7a0d9495101ea1d4a55ab56ba008: Status 404 returned error can't find the container with id 092bb89b928cf128aa231bc081baa685035d7a0d9495101ea1d4a55ab56ba008 Jul 15 16:19:06.089193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:06.089158 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" event={"ID":"e39255c0-38b0-40ed-896f-bf7b212abddd","Type":"ContainerStarted","Data":"092bb89b928cf128aa231bc081baa685035d7a0d9495101ea1d4a55ab56ba008"} Jul 15 16:19:08.098565 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.098530 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" event={"ID":"e39255c0-38b0-40ed-896f-bf7b212abddd","Type":"ContainerStarted","Data":"98eec76e99e1ca4c97b186c4287862c6e37c1de0e1d6585b5e7f339e9d649a6a"} Jul 15 16:19:08.098966 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.098615 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" event={"ID":"e39255c0-38b0-40ed-896f-bf7b212abddd","Type":"ContainerStarted","Data":"5c9cf8908f09eeab3c207c96d7585f635194a6e9ad026eefd7710861429576cc"} Jul 15 16:19:08.098966 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.098635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" event={"ID":"e39255c0-38b0-40ed-896f-bf7b212abddd","Type":"ContainerStarted","Data":"baae20e0ba6cb6a98c068a67dcf54bdd3aa351dc965d508ff7ff3af47bad2145"} Jul 15 16:19:08.123399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.123357 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dc877f5db-wdhq7" podStartSLOduration=1.515315927 podStartE2EDuration="4.123341997s" totalImagesPullingTime="2.60802607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:19:08.12184282 +0000 UTC m=+150.376743044" watchObservedRunningTime="2026-07-15 16:19:08.123341997 +0000 UTC m=+150.378242203" Jul 15 16:19:08.750566 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.750530 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4dfbf65c-skqxr"] Jul 15 16:19:08.752846 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.752830 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.768453 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.768429 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4dfbf65c-skqxr"] Jul 15 16:19:08.841997 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.841954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-service-ca\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.842150 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.842008 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-trusted-ca-bundle\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.842150 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.842086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-serving-cert\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.842150 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.842146 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-oauth-serving-cert\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.842288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.842193 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld7np\" (UniqueName: \"kubernetes.io/projected/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-kube-api-access-ld7np\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.842288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.842251 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-config\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.842363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.842299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-oauth-config\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.943803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.943755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-config\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.943803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.943810 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-oauth-config\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.943850 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-service-ca\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.943893 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-trusted-ca-bundle\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.943920 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-serving-cert\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944291 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.944183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-oauth-serving-cert\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944291 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.944226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld7np\" (UniqueName: \"kubernetes.io/projected/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-kube-api-access-ld7np\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944641 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.944612 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-config\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944766 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.944720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-service-ca\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944852 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.944816 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-trusted-ca-bundle\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.944990 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.944817 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-oauth-serving-cert\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.946534 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.946512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-oauth-config\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.946636 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.946592 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-serving-cert\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:08.953554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:08.953529 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld7np\" (UniqueName: \"kubernetes.io/projected/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-kube-api-access-ld7np\") pod \"console-7d4dfbf65c-skqxr\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:09.063327 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:09.063247 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:09.238687 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:09.238650 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4dfbf65c-skqxr"] Jul 15 16:19:09.252311 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:19:09.252285 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af0642e_b6d8_4f7e_a0ef_5564e52a24b4.slice/crio-9da3e78ba376a2277ca8ae0f1f71929e74d7134ccb947dc8628a168a9b8624f6 WatchSource:0}: Error finding container 9da3e78ba376a2277ca8ae0f1f71929e74d7134ccb947dc8628a168a9b8624f6: Status 404 returned error can't find the container with id 9da3e78ba376a2277ca8ae0f1f71929e74d7134ccb947dc8628a168a9b8624f6 Jul 15 16:19:10.109121 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:10.109086 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4dfbf65c-skqxr" event={"ID":"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4","Type":"ContainerStarted","Data":"d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509"} Jul 15 16:19:10.109293 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:10.109137 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4dfbf65c-skqxr" event={"ID":"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4","Type":"ContainerStarted","Data":"9da3e78ba376a2277ca8ae0f1f71929e74d7134ccb947dc8628a168a9b8624f6"} Jul 15 16:19:10.130152 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:10.130105 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4dfbf65c-skqxr" podStartSLOduration=2.130092246 podStartE2EDuration="2.130092246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:19:10.12908942 +0000 UTC m=+152.383989648" watchObservedRunningTime="2026-07-15 16:19:10.130092246 +0000 UTC m=+152.384992452" Jul 15 16:19:19.063622 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:19.063572 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:19.063622 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:19.063642 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:19.068234 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:19.068217 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:19.148173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:19.148149 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:19:19.199469 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:19.199442 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5bf4994-ls6b9"] Jul 15 16:19:44.221005 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.220965 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5bf4994-ls6b9" podUID="6826515f-c986-431b-b272-4fda2e45e2b1" containerName="console" containerID="cri-o://4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f" gracePeriod=15 Jul 15 16:19:44.455213 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.455192 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:19:44.543105 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543035 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-console-config\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543105 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543093 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-trusted-ca-bundle\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543196 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pkn9\" (UniqueName: \"kubernetes.io/projected/6826515f-c986-431b-b272-4fda2e45e2b1-kube-api-access-8pkn9\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543232 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-serving-cert\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543288 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543277 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-service-ca\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543448 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543305 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-oauth-serving-cert\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543448 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543323 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-oauth-config\") pod \"6826515f-c986-431b-b272-4fda2e45e2b1\" (UID: \"6826515f-c986-431b-b272-4fda2e45e2b1\") " Jul 15 16:19:44.543448 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543376 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-console-config" (OuterVolumeSpecName: "console-config") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:19:44.543588 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543518 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:19:44.543662 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543621 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-console-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:44.543662 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543648 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:19:44.543741 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.543696 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:19:44.545402 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.545377 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6826515f-c986-431b-b272-4fda2e45e2b1-kube-api-access-8pkn9" (OuterVolumeSpecName: "kube-api-access-8pkn9") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "kube-api-access-8pkn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:19:44.545490 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.545412 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:44.545490 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.545441 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6826515f-c986-431b-b272-4fda2e45e2b1" (UID: "6826515f-c986-431b-b272-4fda2e45e2b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:19:44.644188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.644151 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkn9\" (UniqueName: \"kubernetes.io/projected/6826515f-c986-431b-b272-4fda2e45e2b1-kube-api-access-8pkn9\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:44.644188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.644185 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:44.644188 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.644195 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-service-ca\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:44.644365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.644204 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-oauth-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:44.644365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.644212 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6826515f-c986-431b-b272-4fda2e45e2b1-console-oauth-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:44.644365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:44.644220 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6826515f-c986-431b-b272-4fda2e45e2b1-trusted-ca-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:19:45.242744 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.242711 2560 generic.go:356] "Generic (PLEG): container finished" podID="6826515f-c986-431b-b272-4fda2e45e2b1" containerID="4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f" exitCode=0 Jul 15 16:19:45.243221 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.242752 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5bf4994-ls6b9" event={"ID":"6826515f-c986-431b-b272-4fda2e45e2b1","Type":"ContainerDied","Data":"4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f"} Jul 15 16:19:45.243221 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.242761 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5bf4994-ls6b9" Jul 15 16:19:45.243221 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.242790 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5bf4994-ls6b9" event={"ID":"6826515f-c986-431b-b272-4fda2e45e2b1","Type":"ContainerDied","Data":"3f8d48eeef241af3cf7f01649e26c3b7998f0db062cec527e82b7a14377a66e5"} Jul 15 16:19:45.243221 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.242810 2560 scope.go:117] "RemoveContainer" containerID="4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f" Jul 15 16:19:45.251736 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.251717 2560 scope.go:117] "RemoveContainer" containerID="4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f" Jul 15 16:19:45.252045 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:19:45.252024 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f\": container with ID starting with 4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f not found: ID does not exist" containerID="4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f" Jul 15 16:19:45.252124 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.252053 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f"} err="failed to get container status \"4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f\": rpc error: code = NotFound desc = could not find container \"4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f\": container with ID starting with 4e41f693bdce37e0e70fe1fcc06dc6643e2618a7931153bee989c9513508319f not found: ID does not exist" Jul 15 16:19:45.264843 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.264825 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5bf4994-ls6b9"] Jul 15 16:19:45.268740 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:45.268722 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5bf4994-ls6b9"] Jul 15 16:19:46.309057 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:19:46.309026 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6826515f-c986-431b-b272-4fda2e45e2b1" path="/var/lib/kubelet/pods/6826515f-c986-431b-b272-4fda2e45e2b1/volumes" Jul 15 16:20:40.108429 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.108395 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-689bfd8cd-6cfcq"] Jul 15 16:20:40.108885 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.108692 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6826515f-c986-431b-b272-4fda2e45e2b1" containerName="console" Jul 15 16:20:40.108885 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.108703 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6826515f-c986-431b-b272-4fda2e45e2b1" containerName="console" Jul 15 16:20:40.108885 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.108759 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6826515f-c986-431b-b272-4fda2e45e2b1" containerName="console" Jul 15 16:20:40.112927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.112910 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.124088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.124065 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689bfd8cd-6cfcq"] Jul 15 16:20:40.169163 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169138 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-oauth-config\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.169284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-service-ca\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.169284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169208 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-config\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.169364 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169281 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-oauth-serving-cert\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.169364 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169303 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqfn\" (UniqueName: \"kubernetes.io/projected/ba7d89a0-a03b-4efe-bedf-c153168b3b63-kube-api-access-hhqfn\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.169364 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169328 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-serving-cert\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.169470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.169374 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-trusted-ca-bundle\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270513 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270477 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-service-ca\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-config\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270546 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-oauth-serving-cert\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270561 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqfn\" (UniqueName: \"kubernetes.io/projected/ba7d89a0-a03b-4efe-bedf-c153168b3b63-kube-api-access-hhqfn\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270591 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-serving-cert\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-trusted-ca-bundle\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.270671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.270642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-oauth-config\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.271359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.271336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-config\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.271449 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.271361 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-service-ca\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.271449 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.271370 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-oauth-serving-cert\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.271581 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.271561 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-trusted-ca-bundle\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.273245 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.273224 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-oauth-config\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.273295 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.273224 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-serving-cert\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.281321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.281298 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqfn\" (UniqueName: \"kubernetes.io/projected/ba7d89a0-a03b-4efe-bedf-c153168b3b63-kube-api-access-hhqfn\") pod \"console-689bfd8cd-6cfcq\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.421809 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.421751 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:40.546803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:40.546759 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689bfd8cd-6cfcq"] Jul 15 16:20:40.549512 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:20:40.549475 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7d89a0_a03b_4efe_bedf_c153168b3b63.slice/crio-f8a47e23b8a0bf8e5650f1683bf1ef0af974da687562639aff80b9d770f3a5d3 WatchSource:0}: Error finding container f8a47e23b8a0bf8e5650f1683bf1ef0af974da687562639aff80b9d770f3a5d3: Status 404 returned error can't find the container with id f8a47e23b8a0bf8e5650f1683bf1ef0af974da687562639aff80b9d770f3a5d3 Jul 15 16:20:41.449366 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:41.449331 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bfd8cd-6cfcq" event={"ID":"ba7d89a0-a03b-4efe-bedf-c153168b3b63","Type":"ContainerStarted","Data":"12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d"} Jul 15 16:20:41.449757 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:41.449381 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bfd8cd-6cfcq" event={"ID":"ba7d89a0-a03b-4efe-bedf-c153168b3b63","Type":"ContainerStarted","Data":"f8a47e23b8a0bf8e5650f1683bf1ef0af974da687562639aff80b9d770f3a5d3"} Jul 15 16:20:41.467487 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:41.467441 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-689bfd8cd-6cfcq" podStartSLOduration=1.467426704 podStartE2EDuration="1.467426704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:20:41.466852468 +0000 UTC m=+243.721752675" watchObservedRunningTime="2026-07-15 16:20:41.467426704 +0000 UTC m=+243.722326909" Jul 15 16:20:50.422547 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:50.422516 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:50.423054 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:50.422773 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:50.427284 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:50.427264 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:50.483784 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:50.483763 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:20:50.537924 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:20:50.537892 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4dfbf65c-skqxr"] Jul 15 16:21:15.562747 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.562651 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d4dfbf65c-skqxr" podUID="3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" containerName="console" containerID="cri-o://d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509" gracePeriod=15 Jul 15 16:21:15.804291 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.804262 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:21:15.954350 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954321 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld7np\" (UniqueName: \"kubernetes.io/projected/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-kube-api-access-ld7np\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954350 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954353 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-serving-cert\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954544 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954403 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-oauth-serving-cert\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954544 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954426 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-oauth-config\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954544 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954454 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-trusted-ca-bundle\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954544 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954484 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-service-ca\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954544 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954536 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-config\") pod \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\" (UID: \"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4\") " Jul 15 16:21:15.954887 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.954847 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:21:15.955044 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.955014 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:21:15.955044 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.955024 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-service-ca" (OuterVolumeSpecName: "service-ca") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:21:15.955158 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.955116 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-config" (OuterVolumeSpecName: "console-config") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:21:15.956582 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.956559 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-kube-api-access-ld7np" (OuterVolumeSpecName: "kube-api-access-ld7np") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "kube-api-access-ld7np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:21:15.956664 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.956596 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:21:15.956664 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:15.956617 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" (UID: "3af0642e-b6d8-4f7e-a0ef-5564e52a24b4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:21:16.055997 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.055964 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-oauth-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.055997 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.055994 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-oauth-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.055997 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.056003 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-trusted-ca-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.056174 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.056012 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-service-ca\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.056174 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.056021 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.056174 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.056028 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld7np\" (UniqueName: \"kubernetes.io/projected/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-kube-api-access-ld7np\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.056174 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.056036 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4-console-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:21:16.573351 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.573263 2560 generic.go:356] "Generic (PLEG): container finished" podID="3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" containerID="d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509" exitCode=0 Jul 15 16:21:16.573351 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.573311 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4dfbf65c-skqxr" Jul 15 16:21:16.573825 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.573345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4dfbf65c-skqxr" event={"ID":"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4","Type":"ContainerDied","Data":"d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509"} Jul 15 16:21:16.573825 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.573395 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4dfbf65c-skqxr" event={"ID":"3af0642e-b6d8-4f7e-a0ef-5564e52a24b4","Type":"ContainerDied","Data":"9da3e78ba376a2277ca8ae0f1f71929e74d7134ccb947dc8628a168a9b8624f6"} Jul 15 16:21:16.573825 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.573417 2560 scope.go:117] "RemoveContainer" containerID="d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509" Jul 15 16:21:16.581165 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.581153 2560 scope.go:117] "RemoveContainer" containerID="d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509" Jul 15 16:21:16.581404 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:21:16.581368 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509\": container with ID starting with d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509 not found: ID does not exist" containerID="d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509" Jul 15 16:21:16.581446 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.581409 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509"} err="failed to get container status \"d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509\": rpc error: code = NotFound desc = could not find container \"d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509\": container with ID starting with d90397427c95efa8e441a984c088d0c2ab31c520aa97c7ecac98a8b1067d7509 not found: ID does not exist" Jul 15 16:21:16.593323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.593298 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4dfbf65c-skqxr"] Jul 15 16:21:16.594942 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:16.594914 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d4dfbf65c-skqxr"] Jul 15 16:21:18.308423 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:18.308393 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" path="/var/lib/kubelet/pods/3af0642e-b6d8-4f7e-a0ef-5564e52a24b4/volumes" Jul 15 16:21:38.155804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:38.155774 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:21:38.158526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:38.158501 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:21:38.161429 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:38.161410 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 16:21:55.500558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.500520 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz"] Jul 15 16:21:55.502995 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.501004 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" containerName="console" Jul 15 16:21:55.502995 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.501022 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" containerName="console" Jul 15 16:21:55.502995 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.501067 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af0642e-b6d8-4f7e-a0ef-5564e52a24b4" containerName="console" Jul 15 16:21:55.503815 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.503796 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.506933 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.506909 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 16:21:55.507055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.506937 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t4p8z\"" Jul 15 16:21:55.507055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.506950 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 16:21:55.518859 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.518837 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz"] Jul 15 16:21:55.543688 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.543662 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.543800 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.543697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.543800 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.543723 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74fm\" (UniqueName: \"kubernetes.io/projected/4486845c-6c0a-4105-a12f-236fa4d7c4ed-kube-api-access-n74fm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.644100 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.644077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74fm\" (UniqueName: \"kubernetes.io/projected/4486845c-6c0a-4105-a12f-236fa4d7c4ed-kube-api-access-n74fm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.644195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.644131 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.644257 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.644236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.644423 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.644407 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.644500 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.644486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.654765 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.654741 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74fm\" (UniqueName: \"kubernetes.io/projected/4486845c-6c0a-4105-a12f-236fa4d7c4ed-kube-api-access-n74fm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.813006 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.812948 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:21:55.941966 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.941937 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz"] Jul 15 16:21:55.945138 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:21:55.945111 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4486845c_6c0a_4105_a12f_236fa4d7c4ed.slice/crio-9ccc086781015ed201792fe0a5b8751bfc6297c94a3fd5b57927eaa292567f1a WatchSource:0}: Error finding container 9ccc086781015ed201792fe0a5b8751bfc6297c94a3fd5b57927eaa292567f1a: Status 404 returned error can't find the container with id 9ccc086781015ed201792fe0a5b8751bfc6297c94a3fd5b57927eaa292567f1a Jul 15 16:21:55.947055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:55.947039 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 16:21:56.720281 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:21:56.720241 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerStarted","Data":"9ccc086781015ed201792fe0a5b8751bfc6297c94a3fd5b57927eaa292567f1a"} Jul 15 16:22:05.754159 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:05.754117 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerStarted","Data":"15f0da18f41de51d6d27ca7b1b2dbefe560d2d6149e1f1ae7eddb13b3df82075"} Jul 15 16:22:06.759162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:06.759129 2560 generic.go:356] "Generic (PLEG): container finished" podID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerID="15f0da18f41de51d6d27ca7b1b2dbefe560d2d6149e1f1ae7eddb13b3df82075" exitCode=0 Jul 15 16:22:06.759560 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:06.759175 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerDied","Data":"15f0da18f41de51d6d27ca7b1b2dbefe560d2d6149e1f1ae7eddb13b3df82075"} Jul 15 16:22:08.769199 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:08.769173 2560 generic.go:356] "Generic (PLEG): container finished" podID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerID="69784631cb389eb21603dd2e8a727285ffa5b633583b5c97092e2c6bb6844be3" exitCode=0 Jul 15 16:22:08.769516 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:08.769232 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerDied","Data":"69784631cb389eb21603dd2e8a727285ffa5b633583b5c97092e2c6bb6844be3"} Jul 15 16:22:19.816172 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:19.816136 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerStarted","Data":"15221e7eb272dcee641b4a4389b0f0c31e935922819f0e7a4b33ac737f0c2233"} Jul 15 16:22:19.836829 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:19.836785 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" podStartSLOduration=2.38015022 podStartE2EDuration="24.836768552s" totalImagesPullingTime="22.456618332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:21:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:22:19.835312945 +0000 UTC m=+342.090213196" watchObservedRunningTime="2026-07-15 16:22:19.836768552 +0000 UTC m=+342.091668757" Jul 15 16:22:20.822526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:20.822488 2560 generic.go:356] "Generic (PLEG): container finished" podID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerID="15221e7eb272dcee641b4a4389b0f0c31e935922819f0e7a4b33ac737f0c2233" exitCode=0 Jul 15 16:22:20.822975 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:20.822528 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerDied","Data":"15221e7eb272dcee641b4a4389b0f0c31e935922819f0e7a4b33ac737f0c2233"} Jul 15 16:22:21.951211 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:21.951187 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:22:22.071643 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.071609 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-bundle\") pod \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " Jul 15 16:22:22.071796 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.071716 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n74fm\" (UniqueName: \"kubernetes.io/projected/4486845c-6c0a-4105-a12f-236fa4d7c4ed-kube-api-access-n74fm\") pod \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " Jul 15 16:22:22.071796 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.071746 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-util\") pod \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\" (UID: \"4486845c-6c0a-4105-a12f-236fa4d7c4ed\") " Jul 15 16:22:22.072379 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.072355 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-bundle" (OuterVolumeSpecName: "bundle") pod "4486845c-6c0a-4105-a12f-236fa4d7c4ed" (UID: "4486845c-6c0a-4105-a12f-236fa4d7c4ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:22:22.073908 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.073861 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4486845c-6c0a-4105-a12f-236fa4d7c4ed-kube-api-access-n74fm" (OuterVolumeSpecName: "kube-api-access-n74fm") pod "4486845c-6c0a-4105-a12f-236fa4d7c4ed" (UID: "4486845c-6c0a-4105-a12f-236fa4d7c4ed"). InnerVolumeSpecName "kube-api-access-n74fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:22:22.077134 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.077077 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-util" (OuterVolumeSpecName: "util") pod "4486845c-6c0a-4105-a12f-236fa4d7c4ed" (UID: "4486845c-6c0a-4105-a12f-236fa4d7c4ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:22:22.172591 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.172559 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n74fm\" (UniqueName: \"kubernetes.io/projected/4486845c-6c0a-4105-a12f-236fa4d7c4ed-kube-api-access-n74fm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:22.172591 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.172583 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:22.172591 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.172593 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4486845c-6c0a-4105-a12f-236fa4d7c4ed-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:22.831673 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.831645 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" Jul 15 16:22:22.831840 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.831643 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b5qmz" event={"ID":"4486845c-6c0a-4105-a12f-236fa4d7c4ed","Type":"ContainerDied","Data":"9ccc086781015ed201792fe0a5b8751bfc6297c94a3fd5b57927eaa292567f1a"} Jul 15 16:22:22.831840 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:22.831766 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ccc086781015ed201792fe0a5b8751bfc6297c94a3fd5b57927eaa292567f1a" Jul 15 16:22:28.910521 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910492 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65"] Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910784 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="pull" Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910794 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="pull" Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910805 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="extract" Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910810 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="extract" Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910819 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="util" Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910824 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="util" Jul 15 16:22:28.910927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.910886 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4486845c-6c0a-4105-a12f-236fa4d7c4ed" containerName="extract" Jul 15 16:22:28.912802 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.912787 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:28.915476 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.915455 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 16:22:28.915606 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.915501 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 16:22:28.915606 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.915593 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xz2cg\"" Jul 15 16:22:28.927857 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.927826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dch\" (UniqueName: \"kubernetes.io/projected/09c57777-f970-4483-b0f6-38e0dcf85da6-kube-api-access-69dch\") pod \"cert-manager-operator-controller-manager-55c444bf86-88c65\" (UID: \"09c57777-f970-4483-b0f6-38e0dcf85da6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:28.928008 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.927906 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c57777-f970-4483-b0f6-38e0dcf85da6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-88c65\" (UID: \"09c57777-f970-4483-b0f6-38e0dcf85da6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:28.929366 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:28.929339 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65"] Jul 15 16:22:29.029275 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.029238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dch\" (UniqueName: \"kubernetes.io/projected/09c57777-f970-4483-b0f6-38e0dcf85da6-kube-api-access-69dch\") pod \"cert-manager-operator-controller-manager-55c444bf86-88c65\" (UID: \"09c57777-f970-4483-b0f6-38e0dcf85da6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:29.029275 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.029282 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c57777-f970-4483-b0f6-38e0dcf85da6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-88c65\" (UID: \"09c57777-f970-4483-b0f6-38e0dcf85da6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:29.029637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.029622 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c57777-f970-4483-b0f6-38e0dcf85da6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-88c65\" (UID: \"09c57777-f970-4483-b0f6-38e0dcf85da6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:29.040780 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.040755 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dch\" (UniqueName: \"kubernetes.io/projected/09c57777-f970-4483-b0f6-38e0dcf85da6-kube-api-access-69dch\") pod \"cert-manager-operator-controller-manager-55c444bf86-88c65\" (UID: \"09c57777-f970-4483-b0f6-38e0dcf85da6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:29.221897 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.221790 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" Jul 15 16:22:29.364727 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.362289 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65"] Jul 15 16:22:29.367519 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:22:29.367491 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c57777_f970_4483_b0f6_38e0dcf85da6.slice/crio-715137bd60da33267da4d6a92419d54c1e5a73f86d9ba7e5048f4558ea8f5de5 WatchSource:0}: Error finding container 715137bd60da33267da4d6a92419d54c1e5a73f86d9ba7e5048f4558ea8f5de5: Status 404 returned error can't find the container with id 715137bd60da33267da4d6a92419d54c1e5a73f86d9ba7e5048f4558ea8f5de5 Jul 15 16:22:29.859755 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:29.859710 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" event={"ID":"09c57777-f970-4483-b0f6-38e0dcf85da6","Type":"ContainerStarted","Data":"715137bd60da33267da4d6a92419d54c1e5a73f86d9ba7e5048f4558ea8f5de5"} Jul 15 16:22:31.870970 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:31.870920 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" event={"ID":"09c57777-f970-4483-b0f6-38e0dcf85da6","Type":"ContainerStarted","Data":"4e15c2a773b2f214e7d20fe2748a6ecb35c625b011ef33e9316a2c8811931393"} Jul 15 16:22:31.896359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:31.896172 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-88c65" podStartSLOduration=1.590945616 podStartE2EDuration="3.896150884s" totalImagesPullingTime="2.305205268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:22:31.894416266 +0000 UTC m=+354.149316493" watchObservedRunningTime="2026-07-15 16:22:31.896150884 +0000 UTC m=+354.151051090" Jul 15 16:22:32.729569 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.729533 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr4v8"] Jul 15 16:22:32.732070 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.732053 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.735480 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.735458 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 16:22:32.735825 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.735799 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s4mrd\"" Jul 15 16:22:32.736699 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.736681 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 16:22:32.745332 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.745310 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr4v8"] Jul 15 16:22:32.753385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.753355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mrc\" (UniqueName: \"kubernetes.io/projected/cd8f85d5-d999-4730-ac14-e41fd776e3f3-kube-api-access-n5mrc\") pod \"cert-manager-webhook-6486ffd997-jr4v8\" (UID: \"cd8f85d5-d999-4730-ac14-e41fd776e3f3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.753518 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.753442 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd8f85d5-d999-4730-ac14-e41fd776e3f3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr4v8\" (UID: \"cd8f85d5-d999-4730-ac14-e41fd776e3f3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.854097 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.854060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd8f85d5-d999-4730-ac14-e41fd776e3f3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr4v8\" (UID: \"cd8f85d5-d999-4730-ac14-e41fd776e3f3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.854308 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.854170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mrc\" (UniqueName: \"kubernetes.io/projected/cd8f85d5-d999-4730-ac14-e41fd776e3f3-kube-api-access-n5mrc\") pod \"cert-manager-webhook-6486ffd997-jr4v8\" (UID: \"cd8f85d5-d999-4730-ac14-e41fd776e3f3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.866305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.866264 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd8f85d5-d999-4730-ac14-e41fd776e3f3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr4v8\" (UID: \"cd8f85d5-d999-4730-ac14-e41fd776e3f3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.866536 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.866519 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mrc\" (UniqueName: \"kubernetes.io/projected/cd8f85d5-d999-4730-ac14-e41fd776e3f3-kube-api-access-n5mrc\") pod \"cert-manager-webhook-6486ffd997-jr4v8\" (UID: \"cd8f85d5-d999-4730-ac14-e41fd776e3f3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:32.990446 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.990369 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn"] Jul 15 16:22:32.992988 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.992971 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:32.995687 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.995664 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 16:22:32.995806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.995666 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t4p8z\"" Jul 15 16:22:32.995806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:32.995667 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 16:22:33.002205 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.002186 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn"] Jul 15 16:22:33.055492 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.055451 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.055678 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.055509 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.055678 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.055572 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8vz\" (UniqueName: \"kubernetes.io/projected/75d6066e-db3a-49d6-81ed-8d72086b4efc-kube-api-access-cg8vz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.056220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.056200 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:33.156535 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.156496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.156707 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.156575 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.156707 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.156661 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8vz\" (UniqueName: \"kubernetes.io/projected/75d6066e-db3a-49d6-81ed-8d72086b4efc-kube-api-access-cg8vz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.156972 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.156947 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.157085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.156972 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.165299 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.165266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8vz\" (UniqueName: \"kubernetes.io/projected/75d6066e-db3a-49d6-81ed-8d72086b4efc-kube-api-access-cg8vz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.185360 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.185325 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr4v8"] Jul 15 16:22:33.188216 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:22:33.188187 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd8f85d5_d999_4730_ac14_e41fd776e3f3.slice/crio-8b6c8bc902b7dc5a6142b4a2e76f1fb41df129f4c061a6a4e701beb21a489615 WatchSource:0}: Error finding container 8b6c8bc902b7dc5a6142b4a2e76f1fb41df129f4c061a6a4e701beb21a489615: Status 404 returned error can't find the container with id 8b6c8bc902b7dc5a6142b4a2e76f1fb41df129f4c061a6a4e701beb21a489615 Jul 15 16:22:33.303219 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.303179 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:33.448165 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.448136 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn"] Jul 15 16:22:33.449971 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:22:33.449947 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d6066e_db3a_49d6_81ed_8d72086b4efc.slice/crio-74c1bc24cf763d8675044fd2f4ce8a47f6fad1af66ee25944b0a63ad91969c63 WatchSource:0}: Error finding container 74c1bc24cf763d8675044fd2f4ce8a47f6fad1af66ee25944b0a63ad91969c63: Status 404 returned error can't find the container with id 74c1bc24cf763d8675044fd2f4ce8a47f6fad1af66ee25944b0a63ad91969c63 Jul 15 16:22:33.881160 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.881126 2560 generic.go:356] "Generic (PLEG): container finished" podID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerID="9450d3f0e564ecbe11d37eecd8518c7acff0dea65d9793c5b776f46973053d17" exitCode=0 Jul 15 16:22:33.881342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.881209 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" event={"ID":"75d6066e-db3a-49d6-81ed-8d72086b4efc","Type":"ContainerDied","Data":"9450d3f0e564ecbe11d37eecd8518c7acff0dea65d9793c5b776f46973053d17"} Jul 15 16:22:33.881342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.881270 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" event={"ID":"75d6066e-db3a-49d6-81ed-8d72086b4efc","Type":"ContainerStarted","Data":"74c1bc24cf763d8675044fd2f4ce8a47f6fad1af66ee25944b0a63ad91969c63"} Jul 15 16:22:33.882341 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:33.882312 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" event={"ID":"cd8f85d5-d999-4730-ac14-e41fd776e3f3","Type":"ContainerStarted","Data":"8b6c8bc902b7dc5a6142b4a2e76f1fb41df129f4c061a6a4e701beb21a489615"} Jul 15 16:22:35.291890 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.291835 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-24pgk"] Jul 15 16:22:35.295114 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.295088 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.297941 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.297917 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7m47n\"" Jul 15 16:22:35.307229 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.307180 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-24pgk"] Jul 15 16:22:35.374863 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.374828 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9pt\" (UniqueName: \"kubernetes.io/projected/4a353590-3c4f-4e25-950d-d02857ea245b-kube-api-access-jq9pt\") pod \"cert-manager-cainjector-559cdd8c87-24pgk\" (UID: \"4a353590-3c4f-4e25-950d-d02857ea245b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.375071 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.374938 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a353590-3c4f-4e25-950d-d02857ea245b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-24pgk\" (UID: \"4a353590-3c4f-4e25-950d-d02857ea245b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.476135 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.476104 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9pt\" (UniqueName: \"kubernetes.io/projected/4a353590-3c4f-4e25-950d-d02857ea245b-kube-api-access-jq9pt\") pod \"cert-manager-cainjector-559cdd8c87-24pgk\" (UID: \"4a353590-3c4f-4e25-950d-d02857ea245b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.476326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.476206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a353590-3c4f-4e25-950d-d02857ea245b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-24pgk\" (UID: \"4a353590-3c4f-4e25-950d-d02857ea245b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.485839 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.485643 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a353590-3c4f-4e25-950d-d02857ea245b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-24pgk\" (UID: \"4a353590-3c4f-4e25-950d-d02857ea245b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.485839 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.485789 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9pt\" (UniqueName: \"kubernetes.io/projected/4a353590-3c4f-4e25-950d-d02857ea245b-kube-api-access-jq9pt\") pod \"cert-manager-cainjector-559cdd8c87-24pgk\" (UID: \"4a353590-3c4f-4e25-950d-d02857ea245b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.609324 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.609269 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" Jul 15 16:22:35.766419 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.766373 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-24pgk"] Jul 15 16:22:35.769143 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:22:35.769115 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a353590_3c4f_4e25_950d_d02857ea245b.slice/crio-2a0e5ccc4e1b057b7b854c4cafc8beffa3fd8e5be0ac9f2450b4d1a8b35b00ca WatchSource:0}: Error finding container 2a0e5ccc4e1b057b7b854c4cafc8beffa3fd8e5be0ac9f2450b4d1a8b35b00ca: Status 404 returned error can't find the container with id 2a0e5ccc4e1b057b7b854c4cafc8beffa3fd8e5be0ac9f2450b4d1a8b35b00ca Jul 15 16:22:35.894735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:35.894657 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" event={"ID":"4a353590-3c4f-4e25-950d-d02857ea245b","Type":"ContainerStarted","Data":"2a0e5ccc4e1b057b7b854c4cafc8beffa3fd8e5be0ac9f2450b4d1a8b35b00ca"} Jul 15 16:22:37.905165 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.905130 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" event={"ID":"cd8f85d5-d999-4730-ac14-e41fd776e3f3","Type":"ContainerStarted","Data":"f5c4e31189e9215eca31d3ed47b145625263c9767092af13f9404181b7547a87"} Jul 15 16:22:37.905616 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.905306 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:37.906706 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.906678 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" event={"ID":"4a353590-3c4f-4e25-950d-d02857ea245b","Type":"ContainerStarted","Data":"8007109124cf21effc96d32574b9089c30e38778c20e5b7fb0ddc5733b13a83b"} Jul 15 16:22:37.908446 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.908425 2560 generic.go:356] "Generic (PLEG): container finished" podID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerID="7fd6ce21355485343e6708b6e3f5d78816722f18e8228f6cd4a7362df19e6e18" exitCode=0 Jul 15 16:22:37.908562 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.908451 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" event={"ID":"75d6066e-db3a-49d6-81ed-8d72086b4efc","Type":"ContainerDied","Data":"7fd6ce21355485343e6708b6e3f5d78816722f18e8228f6cd4a7362df19e6e18"} Jul 15 16:22:37.924319 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.924270 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" podStartSLOduration=1.57056492 podStartE2EDuration="5.924257581s" totalImagesPullingTime="4.353692661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:22:37.922439101 +0000 UTC m=+360.177339331" watchObservedRunningTime="2026-07-15 16:22:37.924257581 +0000 UTC m=+360.179157787" Jul 15 16:22:37.944300 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:37.944255 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-24pgk" podStartSLOduration=1.172260525 podStartE2EDuration="2.94424075s" totalImagesPullingTime="1.771980225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:22:37.942187102 +0000 UTC m=+360.197087308" watchObservedRunningTime="2026-07-15 16:22:37.94424075 +0000 UTC m=+360.199140956" Jul 15 16:22:38.916411 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:38.916374 2560 generic.go:356] "Generic (PLEG): container finished" podID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerID="3770dd2af644877b1c60542b9db78822708ffe7895c27c84e80bfa71e6a60d78" exitCode=0 Jul 15 16:22:38.916908 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:38.916454 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" event={"ID":"75d6066e-db3a-49d6-81ed-8d72086b4efc","Type":"ContainerDied","Data":"3770dd2af644877b1c60542b9db78822708ffe7895c27c84e80bfa71e6a60d78"} Jul 15 16:22:40.048815 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.048791 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:40.117381 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.117346 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg8vz\" (UniqueName: \"kubernetes.io/projected/75d6066e-db3a-49d6-81ed-8d72086b4efc-kube-api-access-cg8vz\") pod \"75d6066e-db3a-49d6-81ed-8d72086b4efc\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " Jul 15 16:22:40.117538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.117473 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-bundle\") pod \"75d6066e-db3a-49d6-81ed-8d72086b4efc\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " Jul 15 16:22:40.117538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.117498 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-util\") pod \"75d6066e-db3a-49d6-81ed-8d72086b4efc\" (UID: \"75d6066e-db3a-49d6-81ed-8d72086b4efc\") " Jul 15 16:22:40.117863 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.117835 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-bundle" (OuterVolumeSpecName: "bundle") pod "75d6066e-db3a-49d6-81ed-8d72086b4efc" (UID: "75d6066e-db3a-49d6-81ed-8d72086b4efc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:22:40.119475 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.119447 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d6066e-db3a-49d6-81ed-8d72086b4efc-kube-api-access-cg8vz" (OuterVolumeSpecName: "kube-api-access-cg8vz") pod "75d6066e-db3a-49d6-81ed-8d72086b4efc" (UID: "75d6066e-db3a-49d6-81ed-8d72086b4efc"). InnerVolumeSpecName "kube-api-access-cg8vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:22:40.122792 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.122772 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-util" (OuterVolumeSpecName: "util") pod "75d6066e-db3a-49d6-81ed-8d72086b4efc" (UID: "75d6066e-db3a-49d6-81ed-8d72086b4efc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:22:40.218090 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.218002 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:40.218090 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.218029 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d6066e-db3a-49d6-81ed-8d72086b4efc-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:40.218090 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.218042 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg8vz\" (UniqueName: \"kubernetes.io/projected/75d6066e-db3a-49d6-81ed-8d72086b4efc-kube-api-access-cg8vz\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:40.926431 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.926391 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" event={"ID":"75d6066e-db3a-49d6-81ed-8d72086b4efc","Type":"ContainerDied","Data":"74c1bc24cf763d8675044fd2f4ce8a47f6fad1af66ee25944b0a63ad91969c63"} Jul 15 16:22:40.926610 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.926445 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbh8pn" Jul 15 16:22:40.926610 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:40.926449 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c1bc24cf763d8675044fd2f4ce8a47f6fad1af66ee25944b0a63ad91969c63" Jul 15 16:22:43.919634 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:43.919603 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jr4v8" Jul 15 16:22:44.238329 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238247 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6nb4d"] Jul 15 16:22:44.238594 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238580 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="pull" Jul 15 16:22:44.238635 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238596 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="pull" Jul 15 16:22:44.238635 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238626 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="util" Jul 15 16:22:44.238635 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238632 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="util" Jul 15 16:22:44.238735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238642 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="extract" Jul 15 16:22:44.238735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238648 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="extract" Jul 15 16:22:44.238735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.238703 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d6066e-db3a-49d6-81ed-8d72086b4efc" containerName="extract" Jul 15 16:22:44.241976 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.241960 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.244916 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.244899 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ccbtc\"" Jul 15 16:22:44.256057 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.256033 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6nb4d"] Jul 15 16:22:44.349087 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.349058 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8b87\" (UniqueName: \"kubernetes.io/projected/1d01439b-7ef8-4988-97f7-0bc5640ca710-kube-api-access-j8b87\") pod \"cert-manager-55b4d954d5-6nb4d\" (UID: \"1d01439b-7ef8-4988-97f7-0bc5640ca710\") " pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.349233 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.349112 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d01439b-7ef8-4988-97f7-0bc5640ca710-bound-sa-token\") pod \"cert-manager-55b4d954d5-6nb4d\" (UID: \"1d01439b-7ef8-4988-97f7-0bc5640ca710\") " pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.449422 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.449395 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d01439b-7ef8-4988-97f7-0bc5640ca710-bound-sa-token\") pod \"cert-manager-55b4d954d5-6nb4d\" (UID: \"1d01439b-7ef8-4988-97f7-0bc5640ca710\") " pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.449538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.449443 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8b87\" (UniqueName: \"kubernetes.io/projected/1d01439b-7ef8-4988-97f7-0bc5640ca710-kube-api-access-j8b87\") pod \"cert-manager-55b4d954d5-6nb4d\" (UID: \"1d01439b-7ef8-4988-97f7-0bc5640ca710\") " pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.459891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.459854 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d01439b-7ef8-4988-97f7-0bc5640ca710-bound-sa-token\") pod \"cert-manager-55b4d954d5-6nb4d\" (UID: \"1d01439b-7ef8-4988-97f7-0bc5640ca710\") " pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.459995 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.459938 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8b87\" (UniqueName: \"kubernetes.io/projected/1d01439b-7ef8-4988-97f7-0bc5640ca710-kube-api-access-j8b87\") pod \"cert-manager-55b4d954d5-6nb4d\" (UID: \"1d01439b-7ef8-4988-97f7-0bc5640ca710\") " pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.550349 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.550290 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6nb4d" Jul 15 16:22:44.685828 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.685797 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6nb4d"] Jul 15 16:22:44.688790 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:22:44.688764 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d01439b_7ef8_4988_97f7_0bc5640ca710.slice/crio-86ec7d3e913b8a767d1410904e635ff5bd53452226cb71685938779daa757b8e WatchSource:0}: Error finding container 86ec7d3e913b8a767d1410904e635ff5bd53452226cb71685938779daa757b8e: Status 404 returned error can't find the container with id 86ec7d3e913b8a767d1410904e635ff5bd53452226cb71685938779daa757b8e Jul 15 16:22:44.946713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.946676 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6nb4d" event={"ID":"1d01439b-7ef8-4988-97f7-0bc5640ca710","Type":"ContainerStarted","Data":"ed658cf1e43a69ddffcfb70f35acc66c0e4d066b61b7e490d293384692bde095"} Jul 15 16:22:44.947138 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.946722 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6nb4d" event={"ID":"1d01439b-7ef8-4988-97f7-0bc5640ca710","Type":"ContainerStarted","Data":"86ec7d3e913b8a767d1410904e635ff5bd53452226cb71685938779daa757b8e"} Jul 15 16:22:44.967321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:44.967271 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6nb4d" podStartSLOduration=0.967257002 podStartE2EDuration="967.257002ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:22:44.966772245 +0000 UTC m=+367.221672452" watchObservedRunningTime="2026-07-15 16:22:44.967257002 +0000 UTC m=+367.222157208" Jul 15 16:22:52.269137 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.269103 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt"] Jul 15 16:22:52.312383 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.312355 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt"] Jul 15 16:22:52.312528 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.312455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.315936 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.315910 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 16:22:52.317021 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.317004 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 16:22:52.317213 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.317021 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t4p8z\"" Jul 15 16:22:52.412180 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.412155 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.412325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.412271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98j8z\" (UniqueName: \"kubernetes.io/projected/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-kube-api-access-98j8z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.412461 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.412439 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.513652 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.513622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.513789 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.513657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98j8z\" (UniqueName: \"kubernetes.io/projected/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-kube-api-access-98j8z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.513789 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.513695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.514004 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.513988 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.514051 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.514023 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.522745 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.522695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98j8z\" (UniqueName: \"kubernetes.io/projected/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-kube-api-access-98j8z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.622105 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.622083 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:52.765083 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.765059 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt"] Jul 15 16:22:52.767302 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:22:52.767276 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cac7fe1_ce9c_4546_81ac_9eee3f88593d.slice/crio-87840b8bed508f54d9f83a66204aca7816bbf163711039cf42b92aaef23ca4fd WatchSource:0}: Error finding container 87840b8bed508f54d9f83a66204aca7816bbf163711039cf42b92aaef23ca4fd: Status 404 returned error can't find the container with id 87840b8bed508f54d9f83a66204aca7816bbf163711039cf42b92aaef23ca4fd Jul 15 16:22:52.982368 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.982338 2560 generic.go:356] "Generic (PLEG): container finished" podID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerID="db4e3a91e2675486b1bb5c24d8a95c5e6d58c99eb8b681f8f7fb67494ac71b17" exitCode=0 Jul 15 16:22:52.982494 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.982406 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" event={"ID":"4cac7fe1-ce9c-4546-81ac-9eee3f88593d","Type":"ContainerDied","Data":"db4e3a91e2675486b1bb5c24d8a95c5e6d58c99eb8b681f8f7fb67494ac71b17"} Jul 15 16:22:52.982494 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:52.982439 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" event={"ID":"4cac7fe1-ce9c-4546-81ac-9eee3f88593d","Type":"ContainerStarted","Data":"87840b8bed508f54d9f83a66204aca7816bbf163711039cf42b92aaef23ca4fd"} Jul 15 16:22:53.988829 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:53.988800 2560 generic.go:356] "Generic (PLEG): container finished" podID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerID="71c19dd55e90895fd5d92f916c26f29b809610bf00b3b89933d63fa43d5884fc" exitCode=0 Jul 15 16:22:53.989151 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:53.988898 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" event={"ID":"4cac7fe1-ce9c-4546-81ac-9eee3f88593d","Type":"ContainerDied","Data":"71c19dd55e90895fd5d92f916c26f29b809610bf00b3b89933d63fa43d5884fc"} Jul 15 16:22:54.995468 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:54.995434 2560 generic.go:356] "Generic (PLEG): container finished" podID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerID="8090f4902945ea68e843c7bf5bafff97160c130bea1d69775715e079c5e297a3" exitCode=0 Jul 15 16:22:54.995940 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:54.995523 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" event={"ID":"4cac7fe1-ce9c-4546-81ac-9eee3f88593d","Type":"ContainerDied","Data":"8090f4902945ea68e843c7bf5bafff97160c130bea1d69775715e079c5e297a3"} Jul 15 16:22:56.122653 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.122629 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:56.243661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.243630 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-bundle\") pod \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " Jul 15 16:22:56.243818 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.243726 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-util\") pod \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " Jul 15 16:22:56.243818 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.243754 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98j8z\" (UniqueName: \"kubernetes.io/projected/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-kube-api-access-98j8z\") pod \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\" (UID: \"4cac7fe1-ce9c-4546-81ac-9eee3f88593d\") " Jul 15 16:22:56.244430 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.244393 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-bundle" (OuterVolumeSpecName: "bundle") pod "4cac7fe1-ce9c-4546-81ac-9eee3f88593d" (UID: "4cac7fe1-ce9c-4546-81ac-9eee3f88593d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:22:56.245858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.245834 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-kube-api-access-98j8z" (OuterVolumeSpecName: "kube-api-access-98j8z") pod "4cac7fe1-ce9c-4546-81ac-9eee3f88593d" (UID: "4cac7fe1-ce9c-4546-81ac-9eee3f88593d"). InnerVolumeSpecName "kube-api-access-98j8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:22:56.250253 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.250232 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-util" (OuterVolumeSpecName: "util") pod "4cac7fe1-ce9c-4546-81ac-9eee3f88593d" (UID: "4cac7fe1-ce9c-4546-81ac-9eee3f88593d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:22:56.344536 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.344493 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:56.344536 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.344522 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:56.344536 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:56.344536 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98j8z\" (UniqueName: \"kubernetes.io/projected/4cac7fe1-ce9c-4546-81ac-9eee3f88593d-kube-api-access-98j8z\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:22:57.007729 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:57.007698 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" Jul 15 16:22:57.007916 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:57.007699 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft7wqt" event={"ID":"4cac7fe1-ce9c-4546-81ac-9eee3f88593d","Type":"ContainerDied","Data":"87840b8bed508f54d9f83a66204aca7816bbf163711039cf42b92aaef23ca4fd"} Jul 15 16:22:57.007916 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:22:57.007832 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87840b8bed508f54d9f83a66204aca7816bbf163711039cf42b92aaef23ca4fd" Jul 15 16:23:07.411938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.411903 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48"] Jul 15 16:23:07.412326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412233 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="pull" Jul 15 16:23:07.412326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412244 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="pull" Jul 15 16:23:07.412326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412266 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="extract" Jul 15 16:23:07.412326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412271 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="extract" Jul 15 16:23:07.412326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412281 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="util" Jul 15 16:23:07.412326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412287 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="util" Jul 15 16:23:07.412502 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.412335 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cac7fe1-ce9c-4546-81ac-9eee3f88593d" containerName="extract" Jul 15 16:23:07.414915 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.414899 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.418355 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.418335 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 16:23:07.418648 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.418632 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 16:23:07.418701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.418653 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 16:23:07.418701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.418676 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tc9mh\"" Jul 15 16:23:07.419349 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.419333 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 16:23:07.445337 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.445313 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48"] Jul 15 16:23:07.542692 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.542662 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81199833-4935-4991-8d8f-7d86dce289ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.542692 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.542697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81199833-4935-4991-8d8f-7d86dce289ad-manifests\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.542919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.542716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81199833-4935-4991-8d8f-7d86dce289ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.542919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.542739 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbs4r\" (UniqueName: \"kubernetes.io/projected/81199833-4935-4991-8d8f-7d86dce289ad-kube-api-access-wbs4r\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.542919 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.542865 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81199833-4935-4991-8d8f-7d86dce289ad-tmp\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.643304 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.643271 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81199833-4935-4991-8d8f-7d86dce289ad-tmp\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.643460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.643313 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81199833-4935-4991-8d8f-7d86dce289ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.643460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.643332 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81199833-4935-4991-8d8f-7d86dce289ad-manifests\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.643460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.643350 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81199833-4935-4991-8d8f-7d86dce289ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.643460 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.643383 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbs4r\" (UniqueName: \"kubernetes.io/projected/81199833-4935-4991-8d8f-7d86dce289ad-kube-api-access-wbs4r\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.643698 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.643681 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81199833-4935-4991-8d8f-7d86dce289ad-manifests\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.645666 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.645637 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81199833-4935-4991-8d8f-7d86dce289ad-tmp\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.645786 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.645703 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81199833-4935-4991-8d8f-7d86dce289ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.645888 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.645846 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81199833-4935-4991-8d8f-7d86dce289ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.657785 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.657761 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbs4r\" (UniqueName: \"kubernetes.io/projected/81199833-4935-4991-8d8f-7d86dce289ad-kube-api-access-wbs4r\") pod \"opendatahub-operator-controller-manager-599cdff4cb-2bp48\" (UID: \"81199833-4935-4991-8d8f-7d86dce289ad\") " pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.725270 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.725215 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:07.878240 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:07.878208 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48"] Jul 15 16:23:07.881413 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:23:07.881387 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81199833_4935_4991_8d8f_7d86dce289ad.slice/crio-a1d9ae640d502741479db4d6c1eb86c5c54872f8b559fd87b72388aeae801187 WatchSource:0}: Error finding container a1d9ae640d502741479db4d6c1eb86c5c54872f8b559fd87b72388aeae801187: Status 404 returned error can't find the container with id a1d9ae640d502741479db4d6c1eb86c5c54872f8b559fd87b72388aeae801187 Jul 15 16:23:08.056726 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:08.056636 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" event={"ID":"81199833-4935-4991-8d8f-7d86dce289ad","Type":"ContainerStarted","Data":"a1d9ae640d502741479db4d6c1eb86c5c54872f8b559fd87b72388aeae801187"} Jul 15 16:23:12.077934 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:12.077836 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" event={"ID":"81199833-4935-4991-8d8f-7d86dce289ad","Type":"ContainerStarted","Data":"5239cd4680c968e534ec37f2e7d9dcf183e13a3ea8fc4ec5222d30c329732b83"} Jul 15 16:23:13.083312 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:13.083223 2560 generic.go:356] "Generic (PLEG): container finished" podID="81199833-4935-4991-8d8f-7d86dce289ad" containerID="5239cd4680c968e534ec37f2e7d9dcf183e13a3ea8fc4ec5222d30c329732b83" exitCode=0 Jul 15 16:23:13.083684 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:13.083309 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" event={"ID":"81199833-4935-4991-8d8f-7d86dce289ad","Type":"ContainerDied","Data":"5239cd4680c968e534ec37f2e7d9dcf183e13a3ea8fc4ec5222d30c329732b83"} Jul 15 16:23:14.090126 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:14.090089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" event={"ID":"81199833-4935-4991-8d8f-7d86dce289ad","Type":"ContainerStarted","Data":"764bc7c6ae9106cff3e72f9035b898ea91ef2a1b400032a85d2b1fcbbc078857"} Jul 15 16:23:14.090526 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:14.090204 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:14.121406 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:14.121358 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" podStartSLOduration=1.9567458800000002 podStartE2EDuration="7.121341434s" totalImagesPullingTime="4.141352813s" totalInitContainerRuntime="1.023242741s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:23:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:23:14.120299016 +0000 UTC m=+396.375199225" watchObservedRunningTime="2026-07-15 16:23:14.121341434 +0000 UTC m=+396.376241641" Jul 15 16:23:25.097597 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:25.097566 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-599cdff4cb-2bp48" Jul 15 16:23:28.098277 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.098245 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84484457c8-qx84h"] Jul 15 16:23:28.108369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.108348 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.112067 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.112045 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 16:23:28.113084 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.113068 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9vlxs\"" Jul 15 16:23:28.113177 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.113114 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 16:23:28.113177 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.113136 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 16:23:28.113280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.113199 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 16:23:28.115275 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.114975 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bf833c-4532-4ce5-a10a-4d100a580207-tls-certs\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.115275 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.115211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bf833c-4532-4ce5-a10a-4d100a580207-tmp\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.115389 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.115370 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5lz\" (UniqueName: \"kubernetes.io/projected/b0bf833c-4532-4ce5-a10a-4d100a580207-kube-api-access-gg5lz\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.127684 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.127666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84484457c8-qx84h"] Jul 15 16:23:28.216165 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.216130 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bf833c-4532-4ce5-a10a-4d100a580207-tmp\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.216334 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.216183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5lz\" (UniqueName: \"kubernetes.io/projected/b0bf833c-4532-4ce5-a10a-4d100a580207-kube-api-access-gg5lz\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.216334 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.216205 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bf833c-4532-4ce5-a10a-4d100a580207-tls-certs\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.218429 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.218397 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bf833c-4532-4ce5-a10a-4d100a580207-tmp\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.218606 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.218591 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0bf833c-4532-4ce5-a10a-4d100a580207-tls-certs\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.233744 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.233721 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5lz\" (UniqueName: \"kubernetes.io/projected/b0bf833c-4532-4ce5-a10a-4d100a580207-kube-api-access-gg5lz\") pod \"kube-auth-proxy-84484457c8-qx84h\" (UID: \"b0bf833c-4532-4ce5-a10a-4d100a580207\") " pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.326549 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.326522 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r7hzm"] Jul 15 16:23:28.334906 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.334860 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.340862 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.340819 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 16:23:28.341004 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.340913 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qgmzk\"" Jul 15 16:23:28.350688 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.350627 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r7hzm"] Jul 15 16:23:28.417341 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.417308 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx77\" (UniqueName: \"kubernetes.io/projected/3d476544-32be-4eed-8dc5-5db817bc7dd4-kube-api-access-ktx77\") pod \"odh-model-controller-7998c67dc4-r7hzm\" (UID: \"3d476544-32be-4eed-8dc5-5db817bc7dd4\") " pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.417492 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.417364 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d476544-32be-4eed-8dc5-5db817bc7dd4-cert\") pod \"odh-model-controller-7998c67dc4-r7hzm\" (UID: \"3d476544-32be-4eed-8dc5-5db817bc7dd4\") " pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.418162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.418145 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" Jul 15 16:23:28.518170 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.518136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx77\" (UniqueName: \"kubernetes.io/projected/3d476544-32be-4eed-8dc5-5db817bc7dd4-kube-api-access-ktx77\") pod \"odh-model-controller-7998c67dc4-r7hzm\" (UID: \"3d476544-32be-4eed-8dc5-5db817bc7dd4\") " pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.518301 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.518210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d476544-32be-4eed-8dc5-5db817bc7dd4-cert\") pod \"odh-model-controller-7998c67dc4-r7hzm\" (UID: \"3d476544-32be-4eed-8dc5-5db817bc7dd4\") " pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.521399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.521369 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d476544-32be-4eed-8dc5-5db817bc7dd4-cert\") pod \"odh-model-controller-7998c67dc4-r7hzm\" (UID: \"3d476544-32be-4eed-8dc5-5db817bc7dd4\") " pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.549834 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.549807 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx77\" (UniqueName: \"kubernetes.io/projected/3d476544-32be-4eed-8dc5-5db817bc7dd4-kube-api-access-ktx77\") pod \"odh-model-controller-7998c67dc4-r7hzm\" (UID: \"3d476544-32be-4eed-8dc5-5db817bc7dd4\") " pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.608055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.608031 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84484457c8-qx84h"] Jul 15 16:23:28.609553 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:23:28.609526 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bf833c_4532_4ce5_a10a_4d100a580207.slice/crio-6d642e87a06ff8b8e0929fb87b531f29461f485faf1ef835ed9f8b5d6cd65ac7 WatchSource:0}: Error finding container 6d642e87a06ff8b8e0929fb87b531f29461f485faf1ef835ed9f8b5d6cd65ac7: Status 404 returned error can't find the container with id 6d642e87a06ff8b8e0929fb87b531f29461f485faf1ef835ed9f8b5d6cd65ac7 Jul 15 16:23:28.645386 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.645362 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:28.782656 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:28.782622 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r7hzm"] Jul 15 16:23:28.784880 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:23:28.784842 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d476544_32be_4eed_8dc5_5db817bc7dd4.slice/crio-bde09aecac107e98fa568f717d9dadfc5c4fc91ffab79dc7b50a627f7b679b66 WatchSource:0}: Error finding container bde09aecac107e98fa568f717d9dadfc5c4fc91ffab79dc7b50a627f7b679b66: Status 404 returned error can't find the container with id bde09aecac107e98fa568f717d9dadfc5c4fc91ffab79dc7b50a627f7b679b66 Jul 15 16:23:29.160777 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.160740 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" event={"ID":"b0bf833c-4532-4ce5-a10a-4d100a580207","Type":"ContainerStarted","Data":"6d642e87a06ff8b8e0929fb87b531f29461f485faf1ef835ed9f8b5d6cd65ac7"} Jul 15 16:23:29.161850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.161829 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" event={"ID":"3d476544-32be-4eed-8dc5-5db817bc7dd4","Type":"ContainerStarted","Data":"bde09aecac107e98fa568f717d9dadfc5c4fc91ffab79dc7b50a627f7b679b66"} Jul 15 16:23:29.411854 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.411760 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lvlkg"] Jul 15 16:23:29.414626 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.414603 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.417963 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.417940 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 16:23:29.418085 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.417992 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jpnvm\"" Jul 15 16:23:29.425432 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.425406 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60d3362-5052-43eb-8f12-fc6d12f4e1db-tls-certs\") pod \"model-serving-api-54d89bd5fc-lvlkg\" (UID: \"a60d3362-5052-43eb-8f12-fc6d12f4e1db\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.425546 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.425468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbgfs\" (UniqueName: \"kubernetes.io/projected/a60d3362-5052-43eb-8f12-fc6d12f4e1db-kube-api-access-pbgfs\") pod \"model-serving-api-54d89bd5fc-lvlkg\" (UID: \"a60d3362-5052-43eb-8f12-fc6d12f4e1db\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.443290 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.443262 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lvlkg"] Jul 15 16:23:29.526226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.526191 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60d3362-5052-43eb-8f12-fc6d12f4e1db-tls-certs\") pod \"model-serving-api-54d89bd5fc-lvlkg\" (UID: \"a60d3362-5052-43eb-8f12-fc6d12f4e1db\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.526409 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.526270 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbgfs\" (UniqueName: \"kubernetes.io/projected/a60d3362-5052-43eb-8f12-fc6d12f4e1db-kube-api-access-pbgfs\") pod \"model-serving-api-54d89bd5fc-lvlkg\" (UID: \"a60d3362-5052-43eb-8f12-fc6d12f4e1db\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.529196 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.529168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60d3362-5052-43eb-8f12-fc6d12f4e1db-tls-certs\") pod \"model-serving-api-54d89bd5fc-lvlkg\" (UID: \"a60d3362-5052-43eb-8f12-fc6d12f4e1db\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.538412 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.538380 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbgfs\" (UniqueName: \"kubernetes.io/projected/a60d3362-5052-43eb-8f12-fc6d12f4e1db-kube-api-access-pbgfs\") pod \"model-serving-api-54d89bd5fc-lvlkg\" (UID: \"a60d3362-5052-43eb-8f12-fc6d12f4e1db\") " pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.726257 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.725968 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:29.913385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:29.913347 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lvlkg"] Jul 15 16:23:29.917855 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:23:29.917825 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60d3362_5052_43eb_8f12_fc6d12f4e1db.slice/crio-39c514c8f87ef18f945d47e3318d87a780216a3cb7a0abf32d51bc3b8107e4ff WatchSource:0}: Error finding container 39c514c8f87ef18f945d47e3318d87a780216a3cb7a0abf32d51bc3b8107e4ff: Status 404 returned error can't find the container with id 39c514c8f87ef18f945d47e3318d87a780216a3cb7a0abf32d51bc3b8107e4ff Jul 15 16:23:30.169436 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:30.169400 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" event={"ID":"a60d3362-5052-43eb-8f12-fc6d12f4e1db","Type":"ContainerStarted","Data":"39c514c8f87ef18f945d47e3318d87a780216a3cb7a0abf32d51bc3b8107e4ff"} Jul 15 16:23:33.189466 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:33.189433 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" event={"ID":"b0bf833c-4532-4ce5-a10a-4d100a580207","Type":"ContainerStarted","Data":"10ea9df76425e956f01d299efda706a9bb047882858c475d3a7bff92d9486c28"} Jul 15 16:23:33.190968 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:33.190939 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" event={"ID":"3d476544-32be-4eed-8dc5-5db817bc7dd4","Type":"ContainerStarted","Data":"0ab291452b8011b6a43ada2541c3a7f185ed403ba083d744053a33d5ecc3952e"} Jul 15 16:23:33.191100 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:33.191082 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:33.213748 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:33.213704 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84484457c8-qx84h" podStartSLOduration=1.259778024 podStartE2EDuration="5.213687681s" totalImagesPullingTime="3.953909657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:23:33.210499171 +0000 UTC m=+415.465399401" watchObservedRunningTime="2026-07-15 16:23:33.213687681 +0000 UTC m=+415.468588087" Jul 15 16:23:33.233176 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:33.233130 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" podStartSLOduration=1.454688949 podStartE2EDuration="5.233111625s" totalImagesPullingTime="3.778422676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:23:33.231523352 +0000 UTC m=+415.486423573" watchObservedRunningTime="2026-07-15 16:23:33.233111625 +0000 UTC m=+415.488011830" Jul 15 16:23:34.197216 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:34.197176 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" event={"ID":"a60d3362-5052-43eb-8f12-fc6d12f4e1db","Type":"ContainerStarted","Data":"0e6274f5bc09000a348c50d178ac569d0937bd8bbc99c893a78e4f8233b76098"} Jul 15 16:23:34.197631 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:34.197538 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:34.236853 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:34.236801 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" podStartSLOduration=1.944154242 podStartE2EDuration="5.236785314s" totalImagesPullingTime="3.292631072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:23:34.235086632 +0000 UTC m=+416.489986851" watchObservedRunningTime="2026-07-15 16:23:34.236785314 +0000 UTC m=+416.491685520" Jul 15 16:23:35.127391 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.127351 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw"] Jul 15 16:23:35.130735 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.130714 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.134221 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.134197 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 16:23:35.134322 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.134233 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-99pbc\"" Jul 15 16:23:35.153078 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.153050 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw"] Jul 15 16:23:35.178247 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178437 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178258 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178437 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178368 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178437 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178411 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178441 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178467 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178490 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178545 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b172f886-2c3c-4d70-b50d-51f163e507d8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.178580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.178563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhwb\" (UniqueName: \"kubernetes.io/projected/b172f886-2c3c-4d70-b50d-51f163e507d8-kube-api-access-bvhwb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279223 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279187 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b172f886-2c3c-4d70-b50d-51f163e507d8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279223 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279229 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhwb\" (UniqueName: \"kubernetes.io/projected/b172f886-2c3c-4d70-b50d-51f163e507d8-kube-api-access-bvhwb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279285 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279312 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279360 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279389 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279607 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279637 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.279751 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279729 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.280153 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279764 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.280153 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.279946 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.280153 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.280050 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b172f886-2c3c-4d70-b50d-51f163e507d8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.280153 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.280112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.281742 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.281724 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.282081 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.282054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.293666 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.293640 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhwb\" (UniqueName: \"kubernetes.io/projected/b172f886-2c3c-4d70-b50d-51f163e507d8-kube-api-access-bvhwb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.294197 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.294175 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b172f886-2c3c-4d70-b50d-51f163e507d8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pp2zw\" (UID: \"b172f886-2c3c-4d70-b50d-51f163e507d8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.441254 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.441172 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:35.599607 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:35.599568 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw"] Jul 15 16:23:35.603700 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:23:35.603676 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb172f886_2c3c_4d70_b50d_51f163e507d8.slice/crio-594df274af2ef71bde7f7d3d319a4f9cb03be6b9f84a81ba958470282e96e053 WatchSource:0}: Error finding container 594df274af2ef71bde7f7d3d319a4f9cb03be6b9f84a81ba958470282e96e053: Status 404 returned error can't find the container with id 594df274af2ef71bde7f7d3d319a4f9cb03be6b9f84a81ba958470282e96e053 Jul 15 16:23:36.207521 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.207484 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" event={"ID":"b172f886-2c3c-4d70-b50d-51f163e507d8","Type":"ContainerStarted","Data":"594df274af2ef71bde7f7d3d319a4f9cb03be6b9f84a81ba958470282e96e053"} Jul 15 16:23:36.242943 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.242910 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws"] Jul 15 16:23:36.247417 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.247398 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.250738 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.250703 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 16:23:36.250738 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.250720 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 16:23:36.251997 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.251979 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 16:23:36.252742 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.252726 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4frs9\"" Jul 15 16:23:36.252832 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.252785 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 16:23:36.252832 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.252798 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 16:23:36.258329 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.258306 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws"] Jul 15 16:23:36.288551 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.288525 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31af93e2-d84f-4609-b50f-560d6f5ccc0d-metrics-cert\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.288999 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.288565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31af93e2-d84f-4609-b50f-560d6f5ccc0d-manager-config\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.288999 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.288624 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52h87\" (UniqueName: \"kubernetes.io/projected/31af93e2-d84f-4609-b50f-560d6f5ccc0d-kube-api-access-52h87\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.288999 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.288742 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31af93e2-d84f-4609-b50f-560d6f5ccc0d-cert\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.389714 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.389674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31af93e2-d84f-4609-b50f-560d6f5ccc0d-metrics-cert\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.389714 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.389720 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31af93e2-d84f-4609-b50f-560d6f5ccc0d-manager-config\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.389966 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.389755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52h87\" (UniqueName: \"kubernetes.io/projected/31af93e2-d84f-4609-b50f-560d6f5ccc0d-kube-api-access-52h87\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.389966 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.389785 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31af93e2-d84f-4609-b50f-560d6f5ccc0d-cert\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.390508 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.390478 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/31af93e2-d84f-4609-b50f-560d6f5ccc0d-manager-config\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.392277 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.392244 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31af93e2-d84f-4609-b50f-560d6f5ccc0d-metrics-cert\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.392377 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.392297 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31af93e2-d84f-4609-b50f-560d6f5ccc0d-cert\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.399523 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.399501 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52h87\" (UniqueName: \"kubernetes.io/projected/31af93e2-d84f-4609-b50f-560d6f5ccc0d-kube-api-access-52h87\") pod \"lws-controller-manager-5f875d5cd5-wpnws\" (UID: \"31af93e2-d84f-4609-b50f-560d6f5ccc0d\") " pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.558053 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.557964 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:36.712758 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:36.712731 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws"] Jul 15 16:23:36.714930 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:23:36.714903 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31af93e2_d84f_4609_b50f_560d6f5ccc0d.slice/crio-a89c98111abe59d3da523b85e76aae11a2330c32313a596772660094ebb5fff3 WatchSource:0}: Error finding container a89c98111abe59d3da523b85e76aae11a2330c32313a596772660094ebb5fff3: Status 404 returned error can't find the container with id a89c98111abe59d3da523b85e76aae11a2330c32313a596772660094ebb5fff3 Jul 15 16:23:37.215743 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:37.215707 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" event={"ID":"31af93e2-d84f-4609-b50f-560d6f5ccc0d","Type":"ContainerStarted","Data":"a89c98111abe59d3da523b85e76aae11a2330c32313a596772660094ebb5fff3"} Jul 15 16:23:39.164457 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:39.164411 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:23:39.164762 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:39.164518 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:23:39.164762 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:39.164564 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:23:40.235991 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.235951 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" event={"ID":"b172f886-2c3c-4d70-b50d-51f163e507d8","Type":"ContainerStarted","Data":"b79354718bfeb53e015879210b189529d1d94819074d27cd4c51ec5a11888987"} Jul 15 16:23:40.238486 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.238453 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" event={"ID":"31af93e2-d84f-4609-b50f-560d6f5ccc0d","Type":"ContainerStarted","Data":"a35a3dcf4b378b435ef59ed1b5a1d4a115ad738be6882985f822eba9a4cddccd"} Jul 15 16:23:40.238678 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.238658 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:23:40.266574 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.266520 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" podStartSLOduration=1.708089673 podStartE2EDuration="5.266503754s" totalImagesPullingTime="3.558414081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:23:40.265012564 +0000 UTC m=+422.519912795" watchObservedRunningTime="2026-07-15 16:23:40.266503754 +0000 UTC m=+422.521404006" Jul 15 16:23:40.288447 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.288396 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" podStartSLOduration=1.841195177 podStartE2EDuration="4.288378905s" totalImagesPullingTime="2.447183728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:23:40.287310366 +0000 UTC m=+422.542210619" watchObservedRunningTime="2026-07-15 16:23:40.288378905 +0000 UTC m=+422.543279111" Jul 15 16:23:40.441529 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.441497 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:40.446410 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:40.446385 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:41.243902 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:41.243856 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:41.244855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:41.244832 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pp2zw" Jul 15 16:23:44.200321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:44.200294 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r7hzm" Jul 15 16:23:46.211981 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:46.211948 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lvlkg" Jul 15 16:23:51.245835 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:23:51.245798 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f875d5cd5-wpnws" Jul 15 16:24:18.001862 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.001646 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55c8c478b8-zs4z5"] Jul 15 16:24:18.011193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.011163 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.017207 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.017183 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c8c478b8-zs4z5"] Jul 15 16:24:18.056358 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056326 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a78c234c-6a67-425b-bcad-20a77605d0e1-console-serving-cert\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.056358 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056361 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-trusted-ca-bundle\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.056555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056381 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-console-config\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.056555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056404 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-service-ca\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.056555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056425 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-oauth-serving-cert\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.056555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056453 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a78c234c-6a67-425b-bcad-20a77605d0e1-console-oauth-config\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.056555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.056467 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqm2r\" (UniqueName: \"kubernetes.io/projected/a78c234c-6a67-425b-bcad-20a77605d0e1-kube-api-access-wqm2r\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.157841 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a78c234c-6a67-425b-bcad-20a77605d0e1-console-serving-cert\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.157841 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-trusted-ca-bundle\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157859 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-console-config\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157900 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-service-ca\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157920 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-oauth-serving-cert\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157947 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a78c234c-6a67-425b-bcad-20a77605d0e1-console-oauth-config\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.157967 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqm2r\" (UniqueName: \"kubernetes.io/projected/a78c234c-6a67-425b-bcad-20a77605d0e1-kube-api-access-wqm2r\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158764 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.158738 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-service-ca\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158916 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.158744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-console-config\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.158916 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.158802 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-oauth-serving-cert\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.159048 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.158917 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a78c234c-6a67-425b-bcad-20a77605d0e1-trusted-ca-bundle\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.160388 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.160367 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a78c234c-6a67-425b-bcad-20a77605d0e1-console-serving-cert\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.160492 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.160428 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a78c234c-6a67-425b-bcad-20a77605d0e1-console-oauth-config\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.172228 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.172206 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqm2r\" (UniqueName: \"kubernetes.io/projected/a78c234c-6a67-425b-bcad-20a77605d0e1-kube-api-access-wqm2r\") pod \"console-55c8c478b8-zs4z5\" (UID: \"a78c234c-6a67-425b-bcad-20a77605d0e1\") " pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.321291 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.321194 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:18.475009 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:18.474988 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c8c478b8-zs4z5"] Jul 15 16:24:18.477207 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:24:18.477182 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78c234c_6a67_425b_bcad_20a77605d0e1.slice/crio-f97004aacb6722465d52c1c21678427a0d67161dfec23ffc93038bc7a50d02dc WatchSource:0}: Error finding container f97004aacb6722465d52c1c21678427a0d67161dfec23ffc93038bc7a50d02dc: Status 404 returned error can't find the container with id f97004aacb6722465d52c1c21678427a0d67161dfec23ffc93038bc7a50d02dc Jul 15 16:24:19.419912 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:19.419855 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c8c478b8-zs4z5" event={"ID":"a78c234c-6a67-425b-bcad-20a77605d0e1","Type":"ContainerStarted","Data":"d3a81941f2eba534b7f123264aa6d8cc3af423ec909d1d3ba4f6a812c208e5d5"} Jul 15 16:24:19.420376 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:19.419958 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c8c478b8-zs4z5" event={"ID":"a78c234c-6a67-425b-bcad-20a77605d0e1","Type":"ContainerStarted","Data":"f97004aacb6722465d52c1c21678427a0d67161dfec23ffc93038bc7a50d02dc"} Jul 15 16:24:19.441157 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:19.441106 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55c8c478b8-zs4z5" podStartSLOduration=2.441091563 podStartE2EDuration="2.441091563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:24:19.439274173 +0000 UTC m=+461.694174380" watchObservedRunningTime="2026-07-15 16:24:19.441091563 +0000 UTC m=+461.695991770" Jul 15 16:24:28.321346 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:28.321311 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:28.322110 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:28.321372 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:28.326484 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:28.326459 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:28.467730 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:28.467703 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55c8c478b8-zs4z5" Jul 15 16:24:28.517392 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:28.517358 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689bfd8cd-6cfcq"] Jul 15 16:24:53.538432 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.538387 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-689bfd8cd-6cfcq" podUID="ba7d89a0-a03b-4efe-bedf-c153168b3b63" containerName="console" containerID="cri-o://12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d" gracePeriod=15 Jul 15 16:24:53.791422 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.791357 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:24:53.976578 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976546 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhqfn\" (UniqueName: \"kubernetes.io/projected/ba7d89a0-a03b-4efe-bedf-c153168b3b63-kube-api-access-hhqfn\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.976578 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976582 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-serving-cert\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.976793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976643 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-oauth-serving-cert\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.976793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976691 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-oauth-config\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.976793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976716 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-config\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.976793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976767 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-service-ca\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.976793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.976791 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-trusted-ca-bundle\") pod \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\" (UID: \"ba7d89a0-a03b-4efe-bedf-c153168b3b63\") " Jul 15 16:24:53.977133 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.977061 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:24:53.977287 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.977176 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-config" (OuterVolumeSpecName: "console-config") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:24:53.977379 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.977307 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-service-ca" (OuterVolumeSpecName: "service-ca") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:24:53.977425 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.977373 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:24:53.978814 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.978794 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:24:53.979212 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.979190 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:24:53.979297 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:53.979244 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7d89a0-a03b-4efe-bedf-c153168b3b63-kube-api-access-hhqfn" (OuterVolumeSpecName: "kube-api-access-hhqfn") pod "ba7d89a0-a03b-4efe-bedf-c153168b3b63" (UID: "ba7d89a0-a03b-4efe-bedf-c153168b3b63"). InnerVolumeSpecName "kube-api-access-hhqfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:24:54.077713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077620 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-oauth-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.077713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077653 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-config\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.077713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077667 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-service-ca\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.077713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077681 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-trusted-ca-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.077713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077695 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqfn\" (UniqueName: \"kubernetes.io/projected/ba7d89a0-a03b-4efe-bedf-c153168b3b63-kube-api-access-hhqfn\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.077713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077708 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba7d89a0-a03b-4efe-bedf-c153168b3b63-console-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.078054 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.077720 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ba7d89a0-a03b-4efe-bedf-c153168b3b63-oauth-serving-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:24:54.593166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.593130 2560 generic.go:356] "Generic (PLEG): container finished" podID="ba7d89a0-a03b-4efe-bedf-c153168b3b63" containerID="12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d" exitCode=0 Jul 15 16:24:54.593573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.593179 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bfd8cd-6cfcq" Jul 15 16:24:54.593573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.593195 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bfd8cd-6cfcq" event={"ID":"ba7d89a0-a03b-4efe-bedf-c153168b3b63","Type":"ContainerDied","Data":"12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d"} Jul 15 16:24:54.593573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.593245 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bfd8cd-6cfcq" event={"ID":"ba7d89a0-a03b-4efe-bedf-c153168b3b63","Type":"ContainerDied","Data":"f8a47e23b8a0bf8e5650f1683bf1ef0af974da687562639aff80b9d770f3a5d3"} Jul 15 16:24:54.593573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.593331 2560 scope.go:117] "RemoveContainer" containerID="12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d" Jul 15 16:24:54.602144 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.602127 2560 scope.go:117] "RemoveContainer" containerID="12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d" Jul 15 16:24:54.602412 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:24:54.602394 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d\": container with ID starting with 12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d not found: ID does not exist" containerID="12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d" Jul 15 16:24:54.602451 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.602421 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d"} err="failed to get container status \"12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d\": rpc error: code = NotFound desc = could not find container \"12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d\": container with ID starting with 12fbcd3b89f4c94e9e47a3cd598c7871deb787eb3b023a9fd53823e5d5bd6a9d not found: ID does not exist" Jul 15 16:24:54.611556 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.611529 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689bfd8cd-6cfcq"] Jul 15 16:24:54.618850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:54.618824 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-689bfd8cd-6cfcq"] Jul 15 16:24:56.310462 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:24:56.310428 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba7d89a0-a03b-4efe-bedf-c153168b3b63" path="/var/lib/kubelet/pods/ba7d89a0-a03b-4efe-bedf-c153168b3b63/volumes" Jul 15 16:25:00.385731 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.385693 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx"] Jul 15 16:25:00.386325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.386253 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba7d89a0-a03b-4efe-bedf-c153168b3b63" containerName="console" Jul 15 16:25:00.386325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.386272 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba7d89a0-a03b-4efe-bedf-c153168b3b63" containerName="console" Jul 15 16:25:00.386448 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.386380 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba7d89a0-a03b-4efe-bedf-c153168b3b63" containerName="console" Jul 15 16:25:00.391240 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.391221 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.394228 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.394204 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5shnt\"" Jul 15 16:25:00.394350 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.394231 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 16:25:00.396360 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.396341 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 16:25:00.399205 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.399183 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx"] Jul 15 16:25:00.432203 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.432157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8cs\" (UniqueName: \"kubernetes.io/projected/0797ecaa-02dc-4f8a-a374-49ec62364a19-kube-api-access-9k8cs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.432384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.432274 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.432384 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.432310 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.479369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.479337 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r"] Jul 15 16:25:00.482836 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.482820 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.492792 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.492766 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r"] Jul 15 16:25:00.533143 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533110 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.533326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.533326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533189 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.533326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533222 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtc4\" (UniqueName: \"kubernetes.io/projected/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-kube-api-access-mgtc4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.533326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533285 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8cs\" (UniqueName: \"kubernetes.io/projected/0797ecaa-02dc-4f8a-a374-49ec62364a19-kube-api-access-9k8cs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.533501 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533391 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.533501 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533488 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.533583 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.533540 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.542985 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.542955 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8cs\" (UniqueName: \"kubernetes.io/projected/0797ecaa-02dc-4f8a-a374-49ec62364a19-kube-api-access-9k8cs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.633817 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.633778 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.633817 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.633818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.634049 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.633957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtc4\" (UniqueName: \"kubernetes.io/projected/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-kube-api-access-mgtc4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.634217 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.634201 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.634258 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.634227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.643671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.643608 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtc4\" (UniqueName: \"kubernetes.io/projected/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-kube-api-access-mgtc4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.702045 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.702005 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:00.784970 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.784938 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884"] Jul 15 16:25:00.788585 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.788559 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.791991 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.791970 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:00.799327 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.799300 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884"] Jul 15 16:25:00.836103 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.836073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.836261 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.836130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht47n\" (UniqueName: \"kubernetes.io/projected/3e350d22-7902-4628-9cae-469398433e27-kube-api-access-ht47n\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.836308 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.836248 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.848421 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.848387 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx"] Jul 15 16:25:00.852373 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:25:00.852345 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0797ecaa_02dc_4f8a_a374_49ec62364a19.slice/crio-5ff250602203fa45c4988e1818bb057c6a9cefe4d46da5b907c9a33ff378df4d WatchSource:0}: Error finding container 5ff250602203fa45c4988e1818bb057c6a9cefe4d46da5b907c9a33ff378df4d: Status 404 returned error can't find the container with id 5ff250602203fa45c4988e1818bb057c6a9cefe4d46da5b907c9a33ff378df4d Jul 15 16:25:00.937341 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.937307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.937504 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.937388 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht47n\" (UniqueName: \"kubernetes.io/projected/3e350d22-7902-4628-9cae-469398433e27-kube-api-access-ht47n\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.937504 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.937462 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.937692 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.937676 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.937770 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.937752 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:00.942894 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.942858 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r"] Jul 15 16:25:00.944293 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:25:00.944268 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ac79c2_4a48_4b7d_891d_6f706f2a6466.slice/crio-921c521d9478e5321817e95ba2f71f5ac7c45add39acf003808e984a07425020 WatchSource:0}: Error finding container 921c521d9478e5321817e95ba2f71f5ac7c45add39acf003808e984a07425020: Status 404 returned error can't find the container with id 921c521d9478e5321817e95ba2f71f5ac7c45add39acf003808e984a07425020 Jul 15 16:25:00.946672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:00.946650 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht47n\" (UniqueName: \"kubernetes.io/projected/3e350d22-7902-4628-9cae-469398433e27-kube-api-access-ht47n\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:01.102850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.102813 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:01.177199 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.177169 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw"] Jul 15 16:25:01.188073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.188045 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.192546 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.192512 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw"] Jul 15 16:25:01.240136 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.240103 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.240289 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.240158 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.240342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.240298 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklxx\" (UniqueName: \"kubernetes.io/projected/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-kube-api-access-nklxx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.247056 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.246977 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884"] Jul 15 16:25:01.249656 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:25:01.249616 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e350d22_7902_4628_9cae_469398433e27.slice/crio-1e080801eab829485ddecd9e107381635ccd44b7bcb334db0037247bc55d24d9 WatchSource:0}: Error finding container 1e080801eab829485ddecd9e107381635ccd44b7bcb334db0037247bc55d24d9: Status 404 returned error can't find the container with id 1e080801eab829485ddecd9e107381635ccd44b7bcb334db0037247bc55d24d9 Jul 15 16:25:01.340809 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.340774 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nklxx\" (UniqueName: \"kubernetes.io/projected/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-kube-api-access-nklxx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.340998 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.340822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.340998 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.340907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.341278 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.341253 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.341351 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.341262 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.351793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.351770 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklxx\" (UniqueName: \"kubernetes.io/projected/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-kube-api-access-nklxx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.500458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.500356 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:01.631006 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.630985 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw"] Jul 15 16:25:01.631222 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.631200 2560 generic.go:356] "Generic (PLEG): container finished" podID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerID="fa8fd5cae49b331f4cd7460d85b5ca1826082c25a7a0d21cc7cb1b9d82d7e846" exitCode=0 Jul 15 16:25:01.631312 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.631282 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" event={"ID":"0797ecaa-02dc-4f8a-a374-49ec62364a19","Type":"ContainerDied","Data":"fa8fd5cae49b331f4cd7460d85b5ca1826082c25a7a0d21cc7cb1b9d82d7e846"} Jul 15 16:25:01.631363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.631342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" event={"ID":"0797ecaa-02dc-4f8a-a374-49ec62364a19","Type":"ContainerStarted","Data":"5ff250602203fa45c4988e1818bb057c6a9cefe4d46da5b907c9a33ff378df4d"} Jul 15 16:25:01.632968 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.632944 2560 generic.go:356] "Generic (PLEG): container finished" podID="3e350d22-7902-4628-9cae-469398433e27" containerID="649b3833d63beac758e439fea7d65d0b8b41188d84b3d540c1602572cd4ad471" exitCode=0 Jul 15 16:25:01.633086 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.633015 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" event={"ID":"3e350d22-7902-4628-9cae-469398433e27","Type":"ContainerDied","Data":"649b3833d63beac758e439fea7d65d0b8b41188d84b3d540c1602572cd4ad471"} Jul 15 16:25:01.633086 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.633053 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" event={"ID":"3e350d22-7902-4628-9cae-469398433e27","Type":"ContainerStarted","Data":"1e080801eab829485ddecd9e107381635ccd44b7bcb334db0037247bc55d24d9"} Jul 15 16:25:01.633471 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:25:01.633443 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab4cfc6_39e2_46e5_8161_feca59d8b6e0.slice/crio-5c326704df095eb35689183dc19af6fa20881d6b24492f39de4002ea53472962 WatchSource:0}: Error finding container 5c326704df095eb35689183dc19af6fa20881d6b24492f39de4002ea53472962: Status 404 returned error can't find the container with id 5c326704df095eb35689183dc19af6fa20881d6b24492f39de4002ea53472962 Jul 15 16:25:01.634832 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.634774 2560 generic.go:356] "Generic (PLEG): container finished" podID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerID="a371dfe836ab25d8155ca34d9becf4320732a6742eeb479a2641912ab90a0905" exitCode=0 Jul 15 16:25:01.634913 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.634857 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" event={"ID":"c7ac79c2-4a48-4b7d-891d-6f706f2a6466","Type":"ContainerDied","Data":"a371dfe836ab25d8155ca34d9becf4320732a6742eeb479a2641912ab90a0905"} Jul 15 16:25:01.634913 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:01.634907 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" event={"ID":"c7ac79c2-4a48-4b7d-891d-6f706f2a6466","Type":"ContainerStarted","Data":"921c521d9478e5321817e95ba2f71f5ac7c45add39acf003808e984a07425020"} Jul 15 16:25:02.642400 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:02.642366 2560 generic.go:356] "Generic (PLEG): container finished" podID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerID="7d6196d2adf2264d8fedabb3b5e091fecd5b22126ec5070e8702cbae1109c794" exitCode=0 Jul 15 16:25:02.642848 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:02.642460 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" event={"ID":"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0","Type":"ContainerDied","Data":"7d6196d2adf2264d8fedabb3b5e091fecd5b22126ec5070e8702cbae1109c794"} Jul 15 16:25:02.642848 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:02.642510 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" event={"ID":"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0","Type":"ContainerStarted","Data":"5c326704df095eb35689183dc19af6fa20881d6b24492f39de4002ea53472962"} Jul 15 16:25:03.649119 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:03.649092 2560 generic.go:356] "Generic (PLEG): container finished" podID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerID="81c5a2771db529273c3a01af1f12efa1e2b751fec9581a2158651a59fe1432c6" exitCode=0 Jul 15 16:25:03.649554 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:03.649183 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" event={"ID":"0797ecaa-02dc-4f8a-a374-49ec62364a19","Type":"ContainerDied","Data":"81c5a2771db529273c3a01af1f12efa1e2b751fec9581a2158651a59fe1432c6"} Jul 15 16:25:03.651177 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:03.651158 2560 generic.go:356] "Generic (PLEG): container finished" podID="3e350d22-7902-4628-9cae-469398433e27" containerID="5cbb63b99f28728af178a96253119109a4c72c2e584b3bb28158bdf6bc3aac2a" exitCode=0 Jul 15 16:25:03.651290 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:03.651240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" event={"ID":"3e350d22-7902-4628-9cae-469398433e27","Type":"ContainerDied","Data":"5cbb63b99f28728af178a96253119109a4c72c2e584b3bb28158bdf6bc3aac2a"} Jul 15 16:25:03.652756 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:03.652736 2560 generic.go:356] "Generic (PLEG): container finished" podID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerID="881294329b80998d8aca23fae24f71342c6e95f4df8ab71bbde694e68c900aac" exitCode=0 Jul 15 16:25:03.652841 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:03.652806 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" event={"ID":"c7ac79c2-4a48-4b7d-891d-6f706f2a6466","Type":"ContainerDied","Data":"881294329b80998d8aca23fae24f71342c6e95f4df8ab71bbde694e68c900aac"} Jul 15 16:25:04.660185 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.660153 2560 generic.go:356] "Generic (PLEG): container finished" podID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerID="bc49501ac7b95e21939a4b67f837513c436c6e38e202dd2e5180b1fe14fc982b" exitCode=0 Jul 15 16:25:04.660611 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.660230 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" event={"ID":"0797ecaa-02dc-4f8a-a374-49ec62364a19","Type":"ContainerDied","Data":"bc49501ac7b95e21939a4b67f837513c436c6e38e202dd2e5180b1fe14fc982b"} Jul 15 16:25:04.661900 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.661852 2560 generic.go:356] "Generic (PLEG): container finished" podID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerID="ec79b3219f66441b2ff8f6947c1b88ce97a062cd862b91c8cd5cb458861fd5e1" exitCode=0 Jul 15 16:25:04.662015 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.661900 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" event={"ID":"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0","Type":"ContainerDied","Data":"ec79b3219f66441b2ff8f6947c1b88ce97a062cd862b91c8cd5cb458861fd5e1"} Jul 15 16:25:04.664123 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.664103 2560 generic.go:356] "Generic (PLEG): container finished" podID="3e350d22-7902-4628-9cae-469398433e27" containerID="1667005d1bcba2ca07c37aabc80779fd99d4e95c0c10a2ea6a750208bdce6bef" exitCode=0 Jul 15 16:25:04.664210 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.664141 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" event={"ID":"3e350d22-7902-4628-9cae-469398433e27","Type":"ContainerDied","Data":"1667005d1bcba2ca07c37aabc80779fd99d4e95c0c10a2ea6a750208bdce6bef"} Jul 15 16:25:04.666209 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.666191 2560 generic.go:356] "Generic (PLEG): container finished" podID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerID="1c78445b3772c4eacf03d63f6febd0f11517ac9b9c61dffb4db39ed6ae26dfdd" exitCode=0 Jul 15 16:25:04.666292 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:04.666264 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" event={"ID":"c7ac79c2-4a48-4b7d-891d-6f706f2a6466","Type":"ContainerDied","Data":"1c78445b3772c4eacf03d63f6febd0f11517ac9b9c61dffb4db39ed6ae26dfdd"} Jul 15 16:25:05.673938 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.673906 2560 generic.go:356] "Generic (PLEG): container finished" podID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerID="815188ac76a5a194cfd33155e2d62446329c942a8f0c67fd53153e62105dae8e" exitCode=0 Jul 15 16:25:05.674358 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.673992 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" event={"ID":"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0","Type":"ContainerDied","Data":"815188ac76a5a194cfd33155e2d62446329c942a8f0c67fd53153e62105dae8e"} Jul 15 16:25:05.818702 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.818679 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:05.864118 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.864099 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:05.867621 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.867602 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:05.888393 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888373 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-util\") pod \"0797ecaa-02dc-4f8a-a374-49ec62364a19\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " Jul 15 16:25:05.888525 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888420 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgtc4\" (UniqueName: \"kubernetes.io/projected/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-kube-api-access-mgtc4\") pod \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " Jul 15 16:25:05.888525 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888439 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-bundle\") pod \"3e350d22-7902-4628-9cae-469398433e27\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " Jul 15 16:25:05.888525 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888459 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht47n\" (UniqueName: \"kubernetes.io/projected/3e350d22-7902-4628-9cae-469398433e27-kube-api-access-ht47n\") pod \"3e350d22-7902-4628-9cae-469398433e27\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " Jul 15 16:25:05.888525 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888485 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-util\") pod \"3e350d22-7902-4628-9cae-469398433e27\" (UID: \"3e350d22-7902-4628-9cae-469398433e27\") " Jul 15 16:25:05.888729 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888554 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k8cs\" (UniqueName: \"kubernetes.io/projected/0797ecaa-02dc-4f8a-a374-49ec62364a19-kube-api-access-9k8cs\") pod \"0797ecaa-02dc-4f8a-a374-49ec62364a19\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " Jul 15 16:25:05.888972 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.888947 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-util\") pod \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " Jul 15 16:25:05.889362 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.889336 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-bundle" (OuterVolumeSpecName: "bundle") pod "3e350d22-7902-4628-9cae-469398433e27" (UID: "3e350d22-7902-4628-9cae-469398433e27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:05.889458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.889412 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-bundle\") pod \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\" (UID: \"c7ac79c2-4a48-4b7d-891d-6f706f2a6466\") " Jul 15 16:25:05.889513 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.889458 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-bundle\") pod \"0797ecaa-02dc-4f8a-a374-49ec62364a19\" (UID: \"0797ecaa-02dc-4f8a-a374-49ec62364a19\") " Jul 15 16:25:05.890114 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.890090 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-bundle" (OuterVolumeSpecName: "bundle") pod "c7ac79c2-4a48-4b7d-891d-6f706f2a6466" (UID: "c7ac79c2-4a48-4b7d-891d-6f706f2a6466"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:05.890400 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.890384 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.890483 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.890406 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.890781 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.890754 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-bundle" (OuterVolumeSpecName: "bundle") pod "0797ecaa-02dc-4f8a-a374-49ec62364a19" (UID: "0797ecaa-02dc-4f8a-a374-49ec62364a19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:05.892110 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.892084 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-kube-api-access-mgtc4" (OuterVolumeSpecName: "kube-api-access-mgtc4") pod "c7ac79c2-4a48-4b7d-891d-6f706f2a6466" (UID: "c7ac79c2-4a48-4b7d-891d-6f706f2a6466"). InnerVolumeSpecName "kube-api-access-mgtc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:25:05.892352 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.892317 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e350d22-7902-4628-9cae-469398433e27-kube-api-access-ht47n" (OuterVolumeSpecName: "kube-api-access-ht47n") pod "3e350d22-7902-4628-9cae-469398433e27" (UID: "3e350d22-7902-4628-9cae-469398433e27"). InnerVolumeSpecName "kube-api-access-ht47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:25:05.892531 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.892508 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0797ecaa-02dc-4f8a-a374-49ec62364a19-kube-api-access-9k8cs" (OuterVolumeSpecName: "kube-api-access-9k8cs") pod "0797ecaa-02dc-4f8a-a374-49ec62364a19" (UID: "0797ecaa-02dc-4f8a-a374-49ec62364a19"). InnerVolumeSpecName "kube-api-access-9k8cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:25:05.895887 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.895846 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-util" (OuterVolumeSpecName: "util") pod "3e350d22-7902-4628-9cae-469398433e27" (UID: "3e350d22-7902-4628-9cae-469398433e27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:05.895987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.895890 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-util" (OuterVolumeSpecName: "util") pod "c7ac79c2-4a48-4b7d-891d-6f706f2a6466" (UID: "c7ac79c2-4a48-4b7d-891d-6f706f2a6466"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:05.898629 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.898599 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-util" (OuterVolumeSpecName: "util") pod "0797ecaa-02dc-4f8a-a374-49ec62364a19" (UID: "0797ecaa-02dc-4f8a-a374-49ec62364a19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.990967 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.991009 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.991018 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0797ecaa-02dc-4f8a-a374-49ec62364a19-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.991026 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtc4\" (UniqueName: \"kubernetes.io/projected/c7ac79c2-4a48-4b7d-891d-6f706f2a6466-kube-api-access-mgtc4\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.991038 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht47n\" (UniqueName: \"kubernetes.io/projected/3e350d22-7902-4628-9cae-469398433e27-kube-api-access-ht47n\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.991046 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e350d22-7902-4628-9cae-469398433e27-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:05.991063 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:05.991056 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k8cs\" (UniqueName: \"kubernetes.io/projected/0797ecaa-02dc-4f8a-a374-49ec62364a19-kube-api-access-9k8cs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:06.680767 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.680738 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" Jul 15 16:25:06.681247 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.680730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx" event={"ID":"0797ecaa-02dc-4f8a-a374-49ec62364a19","Type":"ContainerDied","Data":"5ff250602203fa45c4988e1818bb057c6a9cefe4d46da5b907c9a33ff378df4d"} Jul 15 16:25:06.681247 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.680912 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff250602203fa45c4988e1818bb057c6a9cefe4d46da5b907c9a33ff378df4d" Jul 15 16:25:06.682803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.682775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" event={"ID":"3e350d22-7902-4628-9cae-469398433e27","Type":"ContainerDied","Data":"1e080801eab829485ddecd9e107381635ccd44b7bcb334db0037247bc55d24d9"} Jul 15 16:25:06.682943 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.682821 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e080801eab829485ddecd9e107381635ccd44b7bcb334db0037247bc55d24d9" Jul 15 16:25:06.683040 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.683021 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884" Jul 15 16:25:06.685044 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.684996 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" Jul 15 16:25:06.685155 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.685051 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r" event={"ID":"c7ac79c2-4a48-4b7d-891d-6f706f2a6466","Type":"ContainerDied","Data":"921c521d9478e5321817e95ba2f71f5ac7c45add39acf003808e984a07425020"} Jul 15 16:25:06.685155 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.685087 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="921c521d9478e5321817e95ba2f71f5ac7c45add39acf003808e984a07425020" Jul 15 16:25:06.800716 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.800695 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:06.897701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.897669 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nklxx\" (UniqueName: \"kubernetes.io/projected/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-kube-api-access-nklxx\") pod \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " Jul 15 16:25:06.897701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.897705 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-bundle\") pod \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " Jul 15 16:25:06.897963 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.897726 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-util\") pod \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\" (UID: \"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0\") " Jul 15 16:25:06.898261 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.898238 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-bundle" (OuterVolumeSpecName: "bundle") pod "1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" (UID: "1ab4cfc6-39e2-46e5-8161-feca59d8b6e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:06.899779 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.899760 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-kube-api-access-nklxx" (OuterVolumeSpecName: "kube-api-access-nklxx") pod "1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" (UID: "1ab4cfc6-39e2-46e5-8161-feca59d8b6e0"). InnerVolumeSpecName "kube-api-access-nklxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:25:06.951162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.951083 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-util" (OuterVolumeSpecName: "util") pod "1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" (UID: "1ab4cfc6-39e2-46e5-8161-feca59d8b6e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:25:06.998338 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.998306 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nklxx\" (UniqueName: \"kubernetes.io/projected/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-kube-api-access-nklxx\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:06.998338 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.998335 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-bundle\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:06.998482 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:06.998344 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ab4cfc6-39e2-46e5-8161-feca59d8b6e0-util\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:25:07.691742 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:07.691711 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" event={"ID":"1ab4cfc6-39e2-46e5-8161-feca59d8b6e0","Type":"ContainerDied","Data":"5c326704df095eb35689183dc19af6fa20881d6b24492f39de4002ea53472962"} Jul 15 16:25:07.691742 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:07.691741 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw" Jul 15 16:25:07.692214 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:07.691762 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c326704df095eb35689183dc19af6fa20881d6b24492f39de4002ea53472962" Jul 15 16:25:27.825949 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.825915 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7"] Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826262 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="util" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826273 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="util" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826279 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="pull" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826284 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="pull" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826292 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="util" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826297 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="util" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826309 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="pull" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826314 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="pull" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826324 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="util" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826329 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="util" Jul 15 16:25:27.826333 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826335 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="util" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826340 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="util" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826347 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826352 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826358 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826363 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826368 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="pull" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826373 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="pull" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826380 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="pull" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826385 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="pull" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826393 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826398 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826408 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826412 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826463 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e350d22-7902-4628-9cae-469398433e27" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826472 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0797ecaa-02dc-4f8a-a374-49ec62364a19" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826479 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab4cfc6-39e2-46e5-8161-feca59d8b6e0" containerName="extract" Jul 15 16:25:27.826658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.826484 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ac79c2-4a48-4b7d-891d-6f706f2a6466" containerName="extract" Jul 15 16:25:27.830099 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.830081 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:27.833362 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.833340 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 16:25:27.833479 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.833408 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 16:25:27.833479 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.833415 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 16:25:27.834566 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.834546 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 16:25:27.834653 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.834546 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5shnt\"" Jul 15 16:25:27.841428 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.841409 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7"] Jul 15 16:25:27.988507 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.988472 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b4b9e23-66e1-47f9-8145-db66dbebb217-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:27.988680 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.988535 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6ft\" (UniqueName: \"kubernetes.io/projected/9b4b9e23-66e1-47f9-8145-db66dbebb217-kube-api-access-zn6ft\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:27.988680 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:27.988619 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4b9e23-66e1-47f9-8145-db66dbebb217-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.089563 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.089529 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b4b9e23-66e1-47f9-8145-db66dbebb217-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.089713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.089593 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ft\" (UniqueName: \"kubernetes.io/projected/9b4b9e23-66e1-47f9-8145-db66dbebb217-kube-api-access-zn6ft\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.089713 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.089632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4b9e23-66e1-47f9-8145-db66dbebb217-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.090253 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.090227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b4b9e23-66e1-47f9-8145-db66dbebb217-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.092001 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.091975 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4b9e23-66e1-47f9-8145-db66dbebb217-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.103202 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.103182 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6ft\" (UniqueName: \"kubernetes.io/projected/9b4b9e23-66e1-47f9-8145-db66dbebb217-kube-api-access-zn6ft\") pod \"kuadrant-console-plugin-66b5774985-rvdb7\" (UID: \"9b4b9e23-66e1-47f9-8145-db66dbebb217\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.139324 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.139292 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" Jul 15 16:25:28.272298 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.272256 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7"] Jul 15 16:25:28.273430 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:25:28.273397 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b4b9e23_66e1_47f9_8145_db66dbebb217.slice/crio-bf9879e91cf38028b1b5c1ed33b9b8bf7497f9def0d851f2c441a0eed7d4e213 WatchSource:0}: Error finding container bf9879e91cf38028b1b5c1ed33b9b8bf7497f9def0d851f2c441a0eed7d4e213: Status 404 returned error can't find the container with id bf9879e91cf38028b1b5c1ed33b9b8bf7497f9def0d851f2c441a0eed7d4e213 Jul 15 16:25:28.800149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:28.800120 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" event={"ID":"9b4b9e23-66e1-47f9-8145-db66dbebb217","Type":"ContainerStarted","Data":"bf9879e91cf38028b1b5c1ed33b9b8bf7497f9def0d851f2c441a0eed7d4e213"} Jul 15 16:25:52.948422 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:52.948386 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" event={"ID":"9b4b9e23-66e1-47f9-8145-db66dbebb217","Type":"ContainerStarted","Data":"ddc388e316fdcc43bcbffdc07f6f188da41dc9c5873fe9083464bd520d7f6665"} Jul 15 16:25:52.968822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:25:52.968771 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvdb7" podStartSLOduration=2.124349416 podStartE2EDuration="25.968755722s" totalImagesPullingTime="23.844406306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:25:52.967914293 +0000 UTC m=+555.222814501" watchObservedRunningTime="2026-07-15 16:25:52.968755722 +0000 UTC m=+555.223655928" Jul 15 16:26:21.738408 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.738373 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-q25nv"] Jul 15 16:26:21.812260 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.812224 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-q25nv"] Jul 15 16:26:21.812420 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.812331 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:21.814948 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.814927 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9xd8j\"" Jul 15 16:26:21.897343 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.897312 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-x26rk"] Jul 15 16:26:21.945300 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.945275 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x26rk"] Jul 15 16:26:21.945440 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.945358 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:21.978104 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:21.978075 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmdd\" (UniqueName: \"kubernetes.io/projected/6158ebee-8858-4af0-899e-dbb878b28873-kube-api-access-ktmdd\") pod \"authorino-58cd955675-q25nv\" (UID: \"6158ebee-8858-4af0-899e-dbb878b28873\") " pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:22.078573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.078500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsck\" (UniqueName: \"kubernetes.io/projected/776897dd-3d8f-4754-b0fa-755b677a28fc-kube-api-access-dnsck\") pod \"authorino-75d556bd77-x26rk\" (UID: \"776897dd-3d8f-4754-b0fa-755b677a28fc\") " pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:22.078573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.078567 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmdd\" (UniqueName: \"kubernetes.io/projected/6158ebee-8858-4af0-899e-dbb878b28873-kube-api-access-ktmdd\") pod \"authorino-58cd955675-q25nv\" (UID: \"6158ebee-8858-4af0-899e-dbb878b28873\") " pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:22.087956 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.087929 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmdd\" (UniqueName: \"kubernetes.io/projected/6158ebee-8858-4af0-899e-dbb878b28873-kube-api-access-ktmdd\") pod \"authorino-58cd955675-q25nv\" (UID: \"6158ebee-8858-4af0-899e-dbb878b28873\") " pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:22.121783 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.121755 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:22.179621 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.179572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsck\" (UniqueName: \"kubernetes.io/projected/776897dd-3d8f-4754-b0fa-755b677a28fc-kube-api-access-dnsck\") pod \"authorino-75d556bd77-x26rk\" (UID: \"776897dd-3d8f-4754-b0fa-755b677a28fc\") " pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:22.193714 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.193672 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsck\" (UniqueName: \"kubernetes.io/projected/776897dd-3d8f-4754-b0fa-755b677a28fc-kube-api-access-dnsck\") pod \"authorino-75d556bd77-x26rk\" (UID: \"776897dd-3d8f-4754-b0fa-755b677a28fc\") " pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:22.254843 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.254814 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:22.278148 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.278123 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-q25nv"] Jul 15 16:26:22.279626 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:26:22.279602 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6158ebee_8858_4af0_899e_dbb878b28873.slice/crio-d10ded2b355a9555dc369186d4c4996d71d9621ecfcfea53d473a11253f775fd WatchSource:0}: Error finding container d10ded2b355a9555dc369186d4c4996d71d9621ecfcfea53d473a11253f775fd: Status 404 returned error can't find the container with id d10ded2b355a9555dc369186d4c4996d71d9621ecfcfea53d473a11253f775fd Jul 15 16:26:22.437369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:22.437344 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x26rk"] Jul 15 16:26:22.439003 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:26:22.438978 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776897dd_3d8f_4754_b0fa_755b677a28fc.slice/crio-b929bedb8a9348b344f971f2c347373781ec1645345efde9bf08bad86bbc2fce WatchSource:0}: Error finding container b929bedb8a9348b344f971f2c347373781ec1645345efde9bf08bad86bbc2fce: Status 404 returned error can't find the container with id b929bedb8a9348b344f971f2c347373781ec1645345efde9bf08bad86bbc2fce Jul 15 16:26:23.118094 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:23.118041 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x26rk" event={"ID":"776897dd-3d8f-4754-b0fa-755b677a28fc","Type":"ContainerStarted","Data":"b929bedb8a9348b344f971f2c347373781ec1645345efde9bf08bad86bbc2fce"} Jul 15 16:26:23.120256 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:23.120178 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q25nv" event={"ID":"6158ebee-8858-4af0-899e-dbb878b28873","Type":"ContainerStarted","Data":"d10ded2b355a9555dc369186d4c4996d71d9621ecfcfea53d473a11253f775fd"} Jul 15 16:26:26.139969 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:26.139846 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x26rk" event={"ID":"776897dd-3d8f-4754-b0fa-755b677a28fc","Type":"ContainerStarted","Data":"8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077"} Jul 15 16:26:26.141459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:26.141433 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q25nv" event={"ID":"6158ebee-8858-4af0-899e-dbb878b28873","Type":"ContainerStarted","Data":"caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0"} Jul 15 16:26:26.157727 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:26.157673 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-x26rk" podStartSLOduration=1.715371675 podStartE2EDuration="5.157657987s" totalImagesPullingTime="3.442286312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:26:26.155766293 +0000 UTC m=+588.410666511" watchObservedRunningTime="2026-07-15 16:26:26.157657987 +0000 UTC m=+588.412558186" Jul 15 16:26:26.173665 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:26.173608 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-q25nv" podStartSLOduration=1.5833589300000002 podStartE2EDuration="5.173588684s" totalImagesPullingTime="3.590229754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:26:26.170798529 +0000 UTC m=+588.425698762" watchObservedRunningTime="2026-07-15 16:26:26.173588684 +0000 UTC m=+588.428488896" Jul 15 16:26:26.192613 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:26.192581 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-q25nv"] Jul 15 16:26:28.150953 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:28.150920 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-q25nv" podUID="6158ebee-8858-4af0-899e-dbb878b28873" containerName="authorino" containerID="cri-o://caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0" gracePeriod=30 Jul 15 16:26:28.397968 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:28.397940 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:28.541193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:28.541112 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktmdd\" (UniqueName: \"kubernetes.io/projected/6158ebee-8858-4af0-899e-dbb878b28873-kube-api-access-ktmdd\") pod \"6158ebee-8858-4af0-899e-dbb878b28873\" (UID: \"6158ebee-8858-4af0-899e-dbb878b28873\") " Jul 15 16:26:28.543177 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:28.543140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6158ebee-8858-4af0-899e-dbb878b28873-kube-api-access-ktmdd" (OuterVolumeSpecName: "kube-api-access-ktmdd") pod "6158ebee-8858-4af0-899e-dbb878b28873" (UID: "6158ebee-8858-4af0-899e-dbb878b28873"). InnerVolumeSpecName "kube-api-access-ktmdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:26:28.641892 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:28.641837 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmdd\" (UniqueName: \"kubernetes.io/projected/6158ebee-8858-4af0-899e-dbb878b28873-kube-api-access-ktmdd\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:26:29.157537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.157504 2560 generic.go:356] "Generic (PLEG): container finished" podID="6158ebee-8858-4af0-899e-dbb878b28873" containerID="caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0" exitCode=0 Jul 15 16:26:29.157973 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.157547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q25nv" event={"ID":"6158ebee-8858-4af0-899e-dbb878b28873","Type":"ContainerDied","Data":"caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0"} Jul 15 16:26:29.157973 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.157549 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q25nv" Jul 15 16:26:29.157973 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.157586 2560 scope.go:117] "RemoveContainer" containerID="caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0" Jul 15 16:26:29.157973 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.157577 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q25nv" event={"ID":"6158ebee-8858-4af0-899e-dbb878b28873","Type":"ContainerDied","Data":"d10ded2b355a9555dc369186d4c4996d71d9621ecfcfea53d473a11253f775fd"} Jul 15 16:26:29.168637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.168618 2560 scope.go:117] "RemoveContainer" containerID="caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0" Jul 15 16:26:29.168929 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:26:29.168907 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0\": container with ID starting with caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0 not found: ID does not exist" containerID="caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0" Jul 15 16:26:29.168987 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.168944 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0"} err="failed to get container status \"caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0\": rpc error: code = NotFound desc = could not find container \"caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0\": container with ID starting with caf689f4ea1ed2b2e711632290c1eb6814e83e100c4843922bd3646517c250d0 not found: ID does not exist" Jul 15 16:26:29.181980 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.181960 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-q25nv"] Jul 15 16:26:29.193006 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:29.192985 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-q25nv"] Jul 15 16:26:30.309860 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:30.309818 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6158ebee-8858-4af0-899e-dbb878b28873" path="/var/lib/kubelet/pods/6158ebee-8858-4af0-899e-dbb878b28873/volumes" Jul 15 16:26:38.195690 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:38.195666 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:26:38.196125 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:38.195844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:26:56.578680 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.578646 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fs2wb"] Jul 15 16:26:56.581065 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.579029 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6158ebee-8858-4af0-899e-dbb878b28873" containerName="authorino" Jul 15 16:26:56.581065 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.579041 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6158ebee-8858-4af0-899e-dbb878b28873" containerName="authorino" Jul 15 16:26:56.581065 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.579101 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6158ebee-8858-4af0-899e-dbb878b28873" containerName="authorino" Jul 15 16:26:56.582061 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.582045 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:56.590647 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.590620 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fs2wb"] Jul 15 16:26:56.687232 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.687196 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xcq\" (UniqueName: \"kubernetes.io/projected/80f141ac-a3bf-42b4-b893-1d1fb9563756-kube-api-access-m8xcq\") pod \"authorino-5d4b67c446-fs2wb\" (UID: \"80f141ac-a3bf-42b4-b893-1d1fb9563756\") " pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:56.788658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.788624 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xcq\" (UniqueName: \"kubernetes.io/projected/80f141ac-a3bf-42b4-b893-1d1fb9563756-kube-api-access-m8xcq\") pod \"authorino-5d4b67c446-fs2wb\" (UID: \"80f141ac-a3bf-42b4-b893-1d1fb9563756\") " pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:56.798036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.798008 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xcq\" (UniqueName: \"kubernetes.io/projected/80f141ac-a3bf-42b4-b893-1d1fb9563756-kube-api-access-m8xcq\") pod \"authorino-5d4b67c446-fs2wb\" (UID: \"80f141ac-a3bf-42b4-b893-1d1fb9563756\") " pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:56.880536 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.880501 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fs2wb"] Jul 15 16:26:56.880787 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:56.880774 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:57.015461 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.015378 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fs2wb"] Jul 15 16:26:57.017924 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:26:57.017893 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f141ac_a3bf_42b4_b893_1d1fb9563756.slice/crio-3a58bb0891bd4a9e8a58cad464e1b990c20fc0725203b606b50287e45ffa126d WatchSource:0}: Error finding container 3a58bb0891bd4a9e8a58cad464e1b990c20fc0725203b606b50287e45ffa126d: Status 404 returned error can't find the container with id 3a58bb0891bd4a9e8a58cad464e1b990c20fc0725203b606b50287e45ffa126d Jul 15 16:26:57.019399 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.019376 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 16:26:57.173795 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.173720 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b86794578-nzgh4"] Jul 15 16:26:57.178595 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.178578 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.181236 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.181210 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 16:26:57.185120 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.185099 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b86794578-nzgh4"] Jul 15 16:26:57.294049 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.294017 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-tls-cert\") pod \"authorino-5b86794578-nzgh4\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.294206 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.294083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvmq\" (UniqueName: \"kubernetes.io/projected/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-kube-api-access-2gvmq\") pod \"authorino-5b86794578-nzgh4\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.303565 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.303534 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" event={"ID":"80f141ac-a3bf-42b4-b893-1d1fb9563756","Type":"ContainerStarted","Data":"3a58bb0891bd4a9e8a58cad464e1b990c20fc0725203b606b50287e45ffa126d"} Jul 15 16:26:57.395662 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.395629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvmq\" (UniqueName: \"kubernetes.io/projected/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-kube-api-access-2gvmq\") pod \"authorino-5b86794578-nzgh4\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.395829 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.395758 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-tls-cert\") pod \"authorino-5b86794578-nzgh4\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.398298 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.398277 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-tls-cert\") pod \"authorino-5b86794578-nzgh4\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.405009 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.404988 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvmq\" (UniqueName: \"kubernetes.io/projected/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-kube-api-access-2gvmq\") pod \"authorino-5b86794578-nzgh4\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.490023 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.489926 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:26:57.619809 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:57.619781 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b86794578-nzgh4"] Jul 15 16:26:57.621114 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:26:57.621083 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3aad75_168a_4f24_89e2_d05c6f5c44a0.slice/crio-597acaac05dc17953714e8612c81733a564ac312bff1f443b8de91af53d71f24 WatchSource:0}: Error finding container 597acaac05dc17953714e8612c81733a564ac312bff1f443b8de91af53d71f24: Status 404 returned error can't find the container with id 597acaac05dc17953714e8612c81733a564ac312bff1f443b8de91af53d71f24 Jul 15 16:26:58.320662 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.320563 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" event={"ID":"80f141ac-a3bf-42b4-b893-1d1fb9563756","Type":"ContainerStarted","Data":"3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a"} Jul 15 16:26:58.320662 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.320597 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" podUID="80f141ac-a3bf-42b4-b893-1d1fb9563756" containerName="authorino" containerID="cri-o://3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a" gracePeriod=30 Jul 15 16:26:58.322128 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.322089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b86794578-nzgh4" event={"ID":"7b3aad75-168a-4f24-89e2-d05c6f5c44a0","Type":"ContainerStarted","Data":"6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af"} Jul 15 16:26:58.322248 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.322135 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b86794578-nzgh4" event={"ID":"7b3aad75-168a-4f24-89e2-d05c6f5c44a0","Type":"ContainerStarted","Data":"597acaac05dc17953714e8612c81733a564ac312bff1f443b8de91af53d71f24"} Jul 15 16:26:58.342418 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.342374 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" podStartSLOduration=2.053667266 podStartE2EDuration="2.342360444s" totalImagesPullingTime="288.693178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:26:58.340948583 +0000 UTC m=+620.595848790" watchObservedRunningTime="2026-07-15 16:26:58.342360444 +0000 UTC m=+620.597260650" Jul 15 16:26:58.361184 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.361127 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b86794578-nzgh4" podStartSLOduration=1.023938555 podStartE2EDuration="1.361109149s" totalImagesPullingTime="337.170594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:26:58.359776105 +0000 UTC m=+620.614676310" watchObservedRunningTime="2026-07-15 16:26:58.361109149 +0000 UTC m=+620.616009356" Jul 15 16:26:58.394362 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.394334 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x26rk"] Jul 15 16:26:58.394513 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.394494 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-x26rk" podUID="776897dd-3d8f-4754-b0fa-755b677a28fc" containerName="authorino" containerID="cri-o://8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077" gracePeriod=30 Jul 15 16:26:58.655727 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.655701 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:58.660488 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.660466 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:58.715397 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.715369 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xcq\" (UniqueName: \"kubernetes.io/projected/80f141ac-a3bf-42b4-b893-1d1fb9563756-kube-api-access-m8xcq\") pod \"80f141ac-a3bf-42b4-b893-1d1fb9563756\" (UID: \"80f141ac-a3bf-42b4-b893-1d1fb9563756\") " Jul 15 16:26:58.717289 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.717268 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f141ac-a3bf-42b4-b893-1d1fb9563756-kube-api-access-m8xcq" (OuterVolumeSpecName: "kube-api-access-m8xcq") pod "80f141ac-a3bf-42b4-b893-1d1fb9563756" (UID: "80f141ac-a3bf-42b4-b893-1d1fb9563756"). InnerVolumeSpecName "kube-api-access-m8xcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:26:58.816126 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.816095 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnsck\" (UniqueName: \"kubernetes.io/projected/776897dd-3d8f-4754-b0fa-755b677a28fc-kube-api-access-dnsck\") pod \"776897dd-3d8f-4754-b0fa-755b677a28fc\" (UID: \"776897dd-3d8f-4754-b0fa-755b677a28fc\") " Jul 15 16:26:58.816294 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.816281 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xcq\" (UniqueName: \"kubernetes.io/projected/80f141ac-a3bf-42b4-b893-1d1fb9563756-kube-api-access-m8xcq\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:26:58.818179 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.818150 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776897dd-3d8f-4754-b0fa-755b677a28fc-kube-api-access-dnsck" (OuterVolumeSpecName: "kube-api-access-dnsck") pod "776897dd-3d8f-4754-b0fa-755b677a28fc" (UID: "776897dd-3d8f-4754-b0fa-755b677a28fc"). InnerVolumeSpecName "kube-api-access-dnsck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:26:58.916924 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:58.916817 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsck\" (UniqueName: \"kubernetes.io/projected/776897dd-3d8f-4754-b0fa-755b677a28fc-kube-api-access-dnsck\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:26:59.328387 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.328263 2560 generic.go:356] "Generic (PLEG): container finished" podID="776897dd-3d8f-4754-b0fa-755b677a28fc" containerID="8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077" exitCode=0 Jul 15 16:26:59.328387 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.328303 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x26rk" Jul 15 16:26:59.328387 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.328342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x26rk" event={"ID":"776897dd-3d8f-4754-b0fa-755b677a28fc","Type":"ContainerDied","Data":"8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077"} Jul 15 16:26:59.328387 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.328382 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x26rk" event={"ID":"776897dd-3d8f-4754-b0fa-755b677a28fc","Type":"ContainerDied","Data":"b929bedb8a9348b344f971f2c347373781ec1645345efde9bf08bad86bbc2fce"} Jul 15 16:26:59.328745 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.328405 2560 scope.go:117] "RemoveContainer" containerID="8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077" Jul 15 16:26:59.329920 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.329893 2560 generic.go:356] "Generic (PLEG): container finished" podID="80f141ac-a3bf-42b4-b893-1d1fb9563756" containerID="3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a" exitCode=0 Jul 15 16:26:59.330036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.329927 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" Jul 15 16:26:59.330036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.329954 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" event={"ID":"80f141ac-a3bf-42b4-b893-1d1fb9563756","Type":"ContainerDied","Data":"3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a"} Jul 15 16:26:59.330036 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.329987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fs2wb" event={"ID":"80f141ac-a3bf-42b4-b893-1d1fb9563756","Type":"ContainerDied","Data":"3a58bb0891bd4a9e8a58cad464e1b990c20fc0725203b606b50287e45ffa126d"} Jul 15 16:26:59.339087 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.339066 2560 scope.go:117] "RemoveContainer" containerID="8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077" Jul 15 16:26:59.339345 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:26:59.339327 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077\": container with ID starting with 8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077 not found: ID does not exist" containerID="8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077" Jul 15 16:26:59.339414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.339354 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077"} err="failed to get container status \"8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077\": rpc error: code = NotFound desc = could not find container \"8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077\": container with ID starting with 8eb5ed34b257527ea58370fb7dfea9f3ac4b2e5b6db40e8aea9eaeccdf0e4077 not found: ID does not exist" Jul 15 16:26:59.339414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.339370 2560 scope.go:117] "RemoveContainer" containerID="3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a" Jul 15 16:26:59.347719 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.347703 2560 scope.go:117] "RemoveContainer" containerID="3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a" Jul 15 16:26:59.347965 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:26:59.347947 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a\": container with ID starting with 3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a not found: ID does not exist" containerID="3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a" Jul 15 16:26:59.348037 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.347973 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a"} err="failed to get container status \"3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a\": rpc error: code = NotFound desc = could not find container \"3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a\": container with ID starting with 3c61fb6e27e5325d7aab745b379744503ea013f18b13b4c317e91e57ce246a3a not found: ID does not exist" Jul 15 16:26:59.369312 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.369292 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x26rk"] Jul 15 16:26:59.371358 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.371340 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x26rk"] Jul 15 16:26:59.408701 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.408681 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fs2wb"] Jul 15 16:26:59.410895 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:26:59.410865 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fs2wb"] Jul 15 16:27:00.311194 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:27:00.311160 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776897dd-3d8f-4754-b0fa-755b677a28fc" path="/var/lib/kubelet/pods/776897dd-3d8f-4754-b0fa-755b677a28fc/volumes" Jul 15 16:27:00.311578 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:27:00.311489 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f141ac-a3bf-42b4-b893-1d1fb9563756" path="/var/lib/kubelet/pods/80f141ac-a3bf-42b4-b893-1d1fb9563756/volumes" Jul 15 16:28:02.514176 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514145 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54"] Jul 15 16:28:02.514555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514502 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776897dd-3d8f-4754-b0fa-755b677a28fc" containerName="authorino" Jul 15 16:28:02.514555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514514 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="776897dd-3d8f-4754-b0fa-755b677a28fc" containerName="authorino" Jul 15 16:28:02.514555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514528 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f141ac-a3bf-42b4-b893-1d1fb9563756" containerName="authorino" Jul 15 16:28:02.514555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514533 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f141ac-a3bf-42b4-b893-1d1fb9563756" containerName="authorino" Jul 15 16:28:02.514688 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514610 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="776897dd-3d8f-4754-b0fa-755b677a28fc" containerName="authorino" Jul 15 16:28:02.514688 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.514618 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f141ac-a3bf-42b4-b893-1d1fb9563756" containerName="authorino" Jul 15 16:28:02.517700 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.517685 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.520571 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.520550 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 16:28:02.520571 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.520561 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cb2r6\"" Jul 15 16:28:02.521823 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.521805 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 16:28:02.521963 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.521936 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 16:28:02.527397 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.527379 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54"] Jul 15 16:28:02.665204 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.665174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.665372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.665229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.665372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.665277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fqns\" (UniqueName: \"kubernetes.io/projected/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-kube-api-access-8fqns\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.665372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.665313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.665372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.665332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.665510 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.665386 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766286 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766346 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fqns\" (UniqueName: \"kubernetes.io/projected/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-kube-api-access-8fqns\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766388 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766659 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766408 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766659 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766426 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766659 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766947 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766923 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.766947 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.766941 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.767087 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.767036 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.768726 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.768708 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.768911 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.768895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.777887 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.777851 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fqns\" (UniqueName: \"kubernetes.io/projected/d0d326ac-6b64-497d-82ca-9b8e98d8aa30-kube-api-access-8fqns\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54\" (UID: \"d0d326ac-6b64-497d-82ca-9b8e98d8aa30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.828627 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.828598 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:02.965517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:02.965491 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54"] Jul 15 16:28:02.969568 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:28:02.969525 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d326ac_6b64_497d_82ca_9b8e98d8aa30.slice/crio-073ead56f4993842f973d3a65f775a502a80d612baa45914929379cb28414ab0 WatchSource:0}: Error finding container 073ead56f4993842f973d3a65f775a502a80d612baa45914929379cb28414ab0: Status 404 returned error can't find the container with id 073ead56f4993842f973d3a65f775a502a80d612baa45914929379cb28414ab0 Jul 15 16:28:03.676464 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:03.676394 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" event={"ID":"d0d326ac-6b64-497d-82ca-9b8e98d8aa30","Type":"ContainerStarted","Data":"073ead56f4993842f973d3a65f775a502a80d612baa45914929379cb28414ab0"} Jul 15 16:28:04.683405 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:04.683364 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" event={"ID":"d0d326ac-6b64-497d-82ca-9b8e98d8aa30","Type":"ContainerStarted","Data":"7e9626e4bcbc615bf9c6a1e4b7e9060c6931a992d095dbb0d5f9c569acda35ed"} Jul 15 16:28:04.704226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:04.704173 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" podStartSLOduration=1.544763007 podStartE2EDuration="2.704157255s" totalImagesPullingTime="1.159394248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:28:04.702539427 +0000 UTC m=+686.957439655" watchObservedRunningTime="2026-07-15 16:28:04.704157255 +0000 UTC m=+686.959057461" Jul 15 16:28:12.829533 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:12.829494 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:12.830013 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:12.829556 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:12.842436 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:12.842416 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:13.741944 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:13.741917 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54" Jul 15 16:28:30.730035 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.729960 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf"] Jul 15 16:28:30.736395 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.736375 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.739008 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.738989 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 16:28:30.745338 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.745315 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf"] Jul 15 16:28:30.809385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.809357 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.809558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.809399 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.809558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.809435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.809558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.809466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.809558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.809514 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fba469-bd61-4909-83a5-d5c61f2f8f4f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.809558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.809531 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcns\" (UniqueName: \"kubernetes.io/projected/29fba469-bd61-4909-83a5-d5c61f2f8f4f-kube-api-access-8lcns\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.909976 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.909941 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.909995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910026 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910045 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910095 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fba469-bd61-4909-83a5-d5c61f2f8f4f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910390 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910233 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcns\" (UniqueName: \"kubernetes.io/projected/29fba469-bd61-4909-83a5-d5c61f2f8f4f-kube-api-access-8lcns\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910451 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910415 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910451 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910418 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.910548 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.910471 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.912850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.912829 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29fba469-bd61-4909-83a5-d5c61f2f8f4f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.913134 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.913117 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fba469-bd61-4909-83a5-d5c61f2f8f4f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:30.919583 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:30.919556 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcns\" (UniqueName: \"kubernetes.io/projected/29fba469-bd61-4909-83a5-d5c61f2f8f4f-kube-api-access-8lcns\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nmxcf\" (UID: \"29fba469-bd61-4909-83a5-d5c61f2f8f4f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:31.048436 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:31.048345 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:31.184944 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:31.184918 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf"] Jul 15 16:28:31.186678 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:28:31.186652 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fba469_bd61_4909_83a5_d5c61f2f8f4f.slice/crio-87a739ae102709e2c8427353c203938d94560204d617a359cfde20a68a91fac1 WatchSource:0}: Error finding container 87a739ae102709e2c8427353c203938d94560204d617a359cfde20a68a91fac1: Status 404 returned error can't find the container with id 87a739ae102709e2c8427353c203938d94560204d617a359cfde20a68a91fac1 Jul 15 16:28:31.828393 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:31.828305 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" event={"ID":"29fba469-bd61-4909-83a5-d5c61f2f8f4f","Type":"ContainerStarted","Data":"06a7135070e1d6f1b66f0c616a3c42ee35152a631bf62e799c10f8abc1271836"} Jul 15 16:28:31.828393 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:31.828374 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" event={"ID":"29fba469-bd61-4909-83a5-d5c61f2f8f4f","Type":"ContainerStarted","Data":"87a739ae102709e2c8427353c203938d94560204d617a359cfde20a68a91fac1"} Jul 15 16:28:31.849080 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:31.849024 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" podStartSLOduration=1.505186021 podStartE2EDuration="1.849006755s" totalImagesPullingTime="343.820734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:28:31.847374144 +0000 UTC m=+714.102274354" watchObservedRunningTime="2026-07-15 16:28:31.849006755 +0000 UTC m=+714.103906960" Jul 15 16:28:41.048663 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:41.048634 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:41.049195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:41.048685 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:41.061093 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:41.061070 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:41.900231 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:41.900205 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nmxcf" Jul 15 16:28:53.024081 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.024024 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j"] Jul 15 16:28:53.026848 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.026829 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.029759 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.029739 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 16:28:53.036858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.036824 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j"] Jul 15 16:28:53.112361 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.112333 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hps75\" (UniqueName: \"kubernetes.io/projected/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-kube-api-access-hps75\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.112468 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.112366 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.112468 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.112395 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.112555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.112473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.112555 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.112517 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.112623 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.112552 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.213675 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.213642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.213822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.213688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.213822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.213730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hps75\" (UniqueName: \"kubernetes.io/projected/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-kube-api-access-hps75\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.213822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.213749 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.213822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.213779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.214073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.213826 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.214073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.214055 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.214187 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.214142 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.214229 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.214186 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.216030 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.216014 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.216201 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.216184 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.223343 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.223321 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hps75\" (UniqueName: \"kubernetes.io/projected/f8fd0bef-97b2-484a-91c5-874b54f8e1aa-kube-api-access-hps75\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j\" (UID: \"f8fd0bef-97b2-484a-91c5-874b54f8e1aa\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.336184 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.336120 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:28:53.468092 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.468061 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j"] Jul 15 16:28:53.469396 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:28:53.469364 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8fd0bef_97b2_484a_91c5_874b54f8e1aa.slice/crio-6328df2e80e360adbfec3c876783366955e4cf452d5c6d68d4520751abbadf48 WatchSource:0}: Error finding container 6328df2e80e360adbfec3c876783366955e4cf452d5c6d68d4520751abbadf48: Status 404 returned error can't find the container with id 6328df2e80e360adbfec3c876783366955e4cf452d5c6d68d4520751abbadf48 Jul 15 16:28:53.957563 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.957481 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" event={"ID":"f8fd0bef-97b2-484a-91c5-874b54f8e1aa","Type":"ContainerStarted","Data":"c46a505a4f5cbed53b218eae1ae4b8860a5b2fa609b7dea6a6a365fcf3715b33"} Jul 15 16:28:53.957563 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.957520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" event={"ID":"f8fd0bef-97b2-484a-91c5-874b54f8e1aa","Type":"ContainerStarted","Data":"6328df2e80e360adbfec3c876783366955e4cf452d5c6d68d4520751abbadf48"} Jul 15 16:28:53.978459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:28:53.978409 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" podStartSLOduration=0.798437121 podStartE2EDuration="978.39407ms" totalImagesPullingTime="179.956949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:28:53.976021444 +0000 UTC m=+736.230921651" watchObservedRunningTime="2026-07-15 16:28:53.97839407 +0000 UTC m=+736.233294276" Jul 15 16:29:03.336244 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:03.336206 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:29:03.336639 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:03.336340 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:29:03.348886 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:03.348837 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:29:04.020135 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:04.020106 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j" Jul 15 16:29:59.877013 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:59.876917 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65c78cc48c-2hz6l"] Jul 15 16:29:59.879725 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:59.879703 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:29:59.889927 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:29:59.889904 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c78cc48c-2hz6l"] Jul 15 16:30:00.004889 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.004831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe-tls-cert\") pod \"authorino-65c78cc48c-2hz6l\" (UID: \"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe\") " pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.005060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.004909 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfwt\" (UniqueName: \"kubernetes.io/projected/6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe-kube-api-access-jvfwt\") pod \"authorino-65c78cc48c-2hz6l\" (UID: \"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe\") " pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.106349 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.106316 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfwt\" (UniqueName: \"kubernetes.io/projected/6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe-kube-api-access-jvfwt\") pod \"authorino-65c78cc48c-2hz6l\" (UID: \"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe\") " pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.106518 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.106415 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe-tls-cert\") pod \"authorino-65c78cc48c-2hz6l\" (UID: \"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe\") " pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.109071 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.109045 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe-tls-cert\") pod \"authorino-65c78cc48c-2hz6l\" (UID: \"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe\") " pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.125016 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.124981 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfwt\" (UniqueName: \"kubernetes.io/projected/6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe-kube-api-access-jvfwt\") pod \"authorino-65c78cc48c-2hz6l\" (UID: \"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe\") " pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.188718 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.188629 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c78cc48c-2hz6l" Jul 15 16:30:00.354122 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:00.354100 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c78cc48c-2hz6l"] Jul 15 16:30:00.356589 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:30:00.356563 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e3f41a8_c4e5_4a1f_b4c7_d0f9cec2a3fe.slice/crio-efb3eb1eaa9deeaac33ed473db1c8542dff77c07d13a4146ad7a45a7c262aa10 WatchSource:0}: Error finding container efb3eb1eaa9deeaac33ed473db1c8542dff77c07d13a4146ad7a45a7c262aa10: Status 404 returned error can't find the container with id efb3eb1eaa9deeaac33ed473db1c8542dff77c07d13a4146ad7a45a7c262aa10 Jul 15 16:30:01.318215 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.318175 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c78cc48c-2hz6l" event={"ID":"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe","Type":"ContainerStarted","Data":"7b59e707f4f60960039cdb0166bbd9434f282513e8a862e1c0a2780fb282ccf5"} Jul 15 16:30:01.318215 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.318215 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c78cc48c-2hz6l" event={"ID":"6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe","Type":"ContainerStarted","Data":"efb3eb1eaa9deeaac33ed473db1c8542dff77c07d13a4146ad7a45a7c262aa10"} Jul 15 16:30:01.347102 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.347040 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65c78cc48c-2hz6l" podStartSLOduration=1.9204101919999998 podStartE2EDuration="2.347017942s" totalImagesPullingTime="426.60775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:30:01.345010652 +0000 UTC m=+803.599910878" watchObservedRunningTime="2026-07-15 16:30:01.347017942 +0000 UTC m=+803.601918149" Jul 15 16:30:01.393891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.392984 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b86794578-nzgh4"] Jul 15 16:30:01.393891 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.393331 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5b86794578-nzgh4" podUID="7b3aad75-168a-4f24-89e2-d05c6f5c44a0" containerName="authorino" containerID="cri-o://6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af" gracePeriod=30 Jul 15 16:30:01.670002 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.667193 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:30:01.821434 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.821342 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-tls-cert\") pod \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " Jul 15 16:30:01.821434 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.821417 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gvmq\" (UniqueName: \"kubernetes.io/projected/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-kube-api-access-2gvmq\") pod \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\" (UID: \"7b3aad75-168a-4f24-89e2-d05c6f5c44a0\") " Jul 15 16:30:01.823537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.823508 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-kube-api-access-2gvmq" (OuterVolumeSpecName: "kube-api-access-2gvmq") pod "7b3aad75-168a-4f24-89e2-d05c6f5c44a0" (UID: "7b3aad75-168a-4f24-89e2-d05c6f5c44a0"). InnerVolumeSpecName "kube-api-access-2gvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:30:01.832257 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.832222 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7b3aad75-168a-4f24-89e2-d05c6f5c44a0" (UID: "7b3aad75-168a-4f24-89e2-d05c6f5c44a0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:30:01.923217 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.923181 2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-tls-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:30:01.923217 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:01.923213 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvmq\" (UniqueName: \"kubernetes.io/projected/7b3aad75-168a-4f24-89e2-d05c6f5c44a0-kube-api-access-2gvmq\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:30:02.324990 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.324953 2560 generic.go:356] "Generic (PLEG): container finished" podID="7b3aad75-168a-4f24-89e2-d05c6f5c44a0" containerID="6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af" exitCode=0 Jul 15 16:30:02.325354 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.325005 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b86794578-nzgh4" Jul 15 16:30:02.325354 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.325027 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b86794578-nzgh4" event={"ID":"7b3aad75-168a-4f24-89e2-d05c6f5c44a0","Type":"ContainerDied","Data":"6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af"} Jul 15 16:30:02.325354 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.325070 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b86794578-nzgh4" event={"ID":"7b3aad75-168a-4f24-89e2-d05c6f5c44a0","Type":"ContainerDied","Data":"597acaac05dc17953714e8612c81733a564ac312bff1f443b8de91af53d71f24"} Jul 15 16:30:02.325354 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.325092 2560 scope.go:117] "RemoveContainer" containerID="6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af" Jul 15 16:30:02.335379 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.335357 2560 scope.go:117] "RemoveContainer" containerID="6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af" Jul 15 16:30:02.335654 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:30:02.335637 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af\": container with ID starting with 6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af not found: ID does not exist" containerID="6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af" Jul 15 16:30:02.335727 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.335662 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af"} err="failed to get container status \"6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af\": rpc error: code = NotFound desc = could not find container \"6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af\": container with ID starting with 6639da14789f114773fa6ffa28e6b3be40f9278955d82043f54482c3e9cab4af not found: ID does not exist" Jul 15 16:30:02.347629 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.347597 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b86794578-nzgh4"] Jul 15 16:30:02.349844 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:02.349820 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b86794578-nzgh4"] Jul 15 16:30:04.311185 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:30:04.311156 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b3aad75-168a-4f24-89e2-d05c6f5c44a0" path="/var/lib/kubelet/pods/7b3aad75-168a-4f24-89e2-d05c6f5c44a0/volumes" Jul 15 16:31:38.238732 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:31:38.238695 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:31:38.239246 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:31:38.239151 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:36:38.284138 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:36:38.284115 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:36:38.287620 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:36:38.287600 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:41:38.326621 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:41:38.326584 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:41:38.330715 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:41:38.330695 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:44:54.031831 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.031800 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g"] Jul 15 16:44:54.032268 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.032217 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3aad75-168a-4f24-89e2-d05c6f5c44a0" containerName="authorino" Jul 15 16:44:54.032268 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.032229 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3aad75-168a-4f24-89e2-d05c6f5c44a0" containerName="authorino" Jul 15 16:44:54.032352 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.032293 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3aad75-168a-4f24-89e2-d05c6f5c44a0" containerName="authorino" Jul 15 16:44:54.035180 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.035163 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.038669 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.038651 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-439e5009-gw-openshift-default-dockercfg-m7xdg\"" Jul 15 16:44:54.050165 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.050140 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g"] Jul 15 16:44:54.100171 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100142 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-data\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100318 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100204 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-envoy\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100318 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100276 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-credential-socket\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100318 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-token\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100428 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100339 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-socket\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100428 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100411 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsl8\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-kube-api-access-jzsl8\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100507 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100443 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-certs\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100507 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100459 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-podinfo\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.100507 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.100487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e66905-5639-4556-bdf6-fb5e1cb08407-istiod-ca-cert\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.201760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201731 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-certs\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.201760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-podinfo\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e66905-5639-4556-bdf6-fb5e1cb08407-istiod-ca-cert\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201816 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-data\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201898 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-envoy\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-credential-socket\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.201973 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-token\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202033 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-socket\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202601 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202079 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsl8\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-kube-api-access-jzsl8\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202601 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202193 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-certs\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202601 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202388 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-credential-socket\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202601 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202456 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-data\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202601 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202589 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-socket\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.202787 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.202614 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e66905-5639-4556-bdf6-fb5e1cb08407-istiod-ca-cert\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.204319 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.204291 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-envoy\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.204447 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.204431 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-podinfo\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.212637 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.212616 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-token\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.212852 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.212837 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsl8\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-kube-api-access-jzsl8\") pod \"e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.346520 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.346501 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:54.477266 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.477236 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g"] Jul 15 16:44:54.478216 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:44:54.478190 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e66905_5639_4556_bdf6_fb5e1cb08407.slice/crio-1509df144c604e7e9b795d62a701f12c94960fa7ac550280b9acad7611a713e7 WatchSource:0}: Error finding container 1509df144c604e7e9b795d62a701f12c94960fa7ac550280b9acad7611a713e7: Status 404 returned error can't find the container with id 1509df144c604e7e9b795d62a701f12c94960fa7ac550280b9acad7611a713e7 Jul 15 16:44:54.480302 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.480273 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:44:54.480389 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.480336 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:44:54.480389 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:54.480363 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:44:55.184922 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:55.184885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" event={"ID":"c5e66905-5639-4556-bdf6-fb5e1cb08407","Type":"ContainerStarted","Data":"12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1"} Jul 15 16:44:55.185373 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:55.184937 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" event={"ID":"c5e66905-5639-4556-bdf6-fb5e1cb08407","Type":"ContainerStarted","Data":"1509df144c604e7e9b795d62a701f12c94960fa7ac550280b9acad7611a713e7"} Jul 15 16:44:55.209686 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:55.209636 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podStartSLOduration=1.209621613 podStartE2EDuration="1.209621613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:44:55.206825948 +0000 UTC m=+1697.461726196" watchObservedRunningTime="2026-07-15 16:44:55.209621613 +0000 UTC m=+1697.464521817" Jul 15 16:44:55.347529 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:55.347491 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:55.352294 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:55.352271 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:56.190695 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:56.190664 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:44:56.191613 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:44:56.191596 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:45:11.191827 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.191788 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 16:45:11.192291 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.191860 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 16:45:11.272730 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.272701 2560 generic.go:356] "Generic (PLEG): container finished" podID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerID="12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1" exitCode=0 Jul 15 16:45:11.272918 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.272772 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" event={"ID":"c5e66905-5639-4556-bdf6-fb5e1cb08407","Type":"ContainerDied","Data":"12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1"} Jul 15 16:45:11.273218 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.273204 2560 scope.go:117] "RemoveContainer" containerID="12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1" Jul 15 16:45:11.274248 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.274220 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:45:11.274325 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.274309 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:45:11.274368 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:11.274353 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:45:12.279681 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:12.279646 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" event={"ID":"c5e66905-5639-4556-bdf6-fb5e1cb08407","Type":"ContainerStarted","Data":"92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7"} Jul 15 16:45:12.347619 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:12.347587 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:45:12.347801 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:12.347779 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 16:45:12.347850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:12.347834 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 16:45:13.348004 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:13.347967 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 16:45:13.348465 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:13.348041 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 16:45:14.346922 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:14.346863 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 16:45:14.347091 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:14.346942 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 16:45:15.347227 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:15.347182 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 16:45:15.347750 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:15.347249 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 16:45:16.346932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:16.346893 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 15 16:45:16.347146 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:16.346974 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 15 16:45:17.351315 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:17.351281 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:45:17.351822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:17.351651 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:45:17.352339 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:17.352324 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:45:59.992895 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:59.992842 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g"] Jul 15 16:45:59.993650 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:45:59.993064 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" containerID="cri-o://92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7" gracePeriod=30 Jul 15 16:46:02.354391 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:02.354356 2560 patch_prober.go:28] interesting pod/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 16:46:02.354763 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:02.354422 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 16:46:05.240197 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.240167 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:46:05.285322 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285231 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e66905-5639-4556-bdf6-fb5e1cb08407-istiod-ca-cert\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285322 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzsl8\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-kube-api-access-jzsl8\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285548 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285327 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-credential-socket\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285548 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285352 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-socket\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285548 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285424 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-envoy\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285548 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285449 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-podinfo\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285548 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285544 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-data\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285810 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285568 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-certs\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285810 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285644 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-token\") pod \"c5e66905-5639-4556-bdf6-fb5e1cb08407\" (UID: \"c5e66905-5639-4556-bdf6-fb5e1cb08407\") " Jul 15 16:46:05.285810 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285676 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:46:05.285810 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285685 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e66905-5639-4556-bdf6-fb5e1cb08407-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:46:05.286137 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285937 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e66905-5639-4556-bdf6-fb5e1cb08407-istiod-ca-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.286137 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.285954 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-credential-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.286276 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.286247 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:46:05.286418 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.286380 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:46:05.288308 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.288271 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-kube-api-access-jzsl8" (OuterVolumeSpecName: "kube-api-access-jzsl8") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "kube-api-access-jzsl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:46:05.288541 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.288516 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 16:46:05.288641 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.288624 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:46:05.288718 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.288694 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:46:05.288803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.288789 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5e66905-5639-4556-bdf6-fb5e1cb08407" (UID: "c5e66905-5639-4556-bdf6-fb5e1cb08407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:46:05.386647 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386617 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-token\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.386647 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386645 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzsl8\" (UniqueName: \"kubernetes.io/projected/c5e66905-5639-4556-bdf6-fb5e1cb08407-kube-api-access-jzsl8\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.386647 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386654 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.386861 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386663 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-envoy\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.386861 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386673 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-podinfo\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.386861 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386682 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-istio-data\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.386861 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.386690 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e66905-5639-4556-bdf6-fb5e1cb08407-workload-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:46:05.571195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.571097 2560 generic.go:356] "Generic (PLEG): container finished" podID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerID="92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7" exitCode=0 Jul 15 16:46:05.571195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.571150 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" event={"ID":"c5e66905-5639-4556-bdf6-fb5e1cb08407","Type":"ContainerDied","Data":"92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7"} Jul 15 16:46:05.571195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.571191 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" event={"ID":"c5e66905-5639-4556-bdf6-fb5e1cb08407","Type":"ContainerDied","Data":"1509df144c604e7e9b795d62a701f12c94960fa7ac550280b9acad7611a713e7"} Jul 15 16:46:05.571478 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.571213 2560 scope.go:117] "RemoveContainer" containerID="92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7" Jul 15 16:46:05.571478 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.571152 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g" Jul 15 16:46:05.580932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.580909 2560 scope.go:117] "RemoveContainer" containerID="12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1" Jul 15 16:46:05.588855 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.588835 2560 scope.go:117] "RemoveContainer" containerID="92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7" Jul 15 16:46:05.589134 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:46:05.589114 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7\": container with ID starting with 92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7 not found: ID does not exist" containerID="92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7" Jul 15 16:46:05.589193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.589143 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7"} err="failed to get container status \"92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7\": rpc error: code = NotFound desc = could not find container \"92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7\": container with ID starting with 92c46a5aee7a88fbc3c6ca4488644e8b07898f39c2b24d575d954a7d842fa7d7 not found: ID does not exist" Jul 15 16:46:05.589193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.589160 2560 scope.go:117] "RemoveContainer" containerID="12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1" Jul 15 16:46:05.589356 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:46:05.589340 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1\": container with ID starting with 12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1 not found: ID does not exist" containerID="12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1" Jul 15 16:46:05.589398 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.589361 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1"} err="failed to get container status \"12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1\": rpc error: code = NotFound desc = could not find container \"12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1\": container with ID starting with 12e58594b39da3ff29c8fdd1edfee9aba4b0abda7622ff835c49612f003848b1 not found: ID does not exist" Jul 15 16:46:05.595357 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.595335 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g"] Jul 15 16:46:05.597576 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:05.597553 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-439e5009-gw-openshift-default-7f55db565f-twp2g"] Jul 15 16:46:06.311222 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:06.311181 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" path="/var/lib/kubelet/pods/c5e66905-5639-4556-bdf6-fb5e1cb08407/volumes" Jul 15 16:46:38.361721 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:38.361693 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:46:38.366821 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:46:38.366800 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:47:29.332092 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.332060 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5"] Jul 15 16:47:29.332569 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.332552 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" Jul 15 16:47:29.332569 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.332566 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" Jul 15 16:47:29.332675 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.332580 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" Jul 15 16:47:29.332675 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.332586 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" Jul 15 16:47:29.332675 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.332647 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" Jul 15 16:47:29.335463 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.335440 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.338356 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.338331 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-77b034d1-openshift-default-dockercfg-s79vz\"" Jul 15 16:47:29.354834 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.354799 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5"] Jul 15 16:47:29.459576 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459520 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-certs\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459576 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqkt\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-kube-api-access-rmqkt\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459623 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-envoy\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-socket\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459691 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-podinfo\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-token\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-data\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459803 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-credential-socket\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.459851 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.459823 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istiod-ca-cert\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.560594 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560555 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-data\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.560806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560616 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-credential-socket\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.560806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560647 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istiod-ca-cert\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.560806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-certs\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.560806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560728 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqkt\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-kube-api-access-rmqkt\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.560806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-envoy\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-socket\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-podinfo\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.560931 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-token\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.561060 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-data\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561169 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.561101 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-credential-socket\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561388 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.561209 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-socket\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561388 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.561246 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-certs\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.561506 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.561485 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istiod-ca-cert\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.563303 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.563279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-envoy\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.563503 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.563485 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-podinfo\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.569817 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.569790 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-token\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.569969 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.569952 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqkt\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-kube-api-access-rmqkt\") pod \"e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.648336 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.648305 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:29.796985 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.796953 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5"] Jul 15 16:47:29.797902 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:47:29.797858 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9918a6bd_b92f_4b1b_8f0c_87b04fe1de7d.slice/crio-37291e6da9e2ca36d34abffa64b66f06bd83ede2ffb46ce5cb5a744994db52e1 WatchSource:0}: Error finding container 37291e6da9e2ca36d34abffa64b66f06bd83ede2ffb46ce5cb5a744994db52e1: Status 404 returned error can't find the container with id 37291e6da9e2ca36d34abffa64b66f06bd83ede2ffb46ce5cb5a744994db52e1 Jul 15 16:47:29.800060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.800033 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:47:29.800137 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.800093 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:47:29.800137 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:29.800120 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:47:30.022748 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:30.022446 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" event={"ID":"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d","Type":"ContainerStarted","Data":"4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769"} Jul 15 16:47:30.022748 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:30.022513 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" event={"ID":"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d","Type":"ContainerStarted","Data":"37291e6da9e2ca36d34abffa64b66f06bd83ede2ffb46ce5cb5a744994db52e1"} Jul 15 16:47:30.045189 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:30.045133 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podStartSLOduration=1.045111131 podStartE2EDuration="1.045111131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:47:30.043775246 +0000 UTC m=+1852.298675453" watchObservedRunningTime="2026-07-15 16:47:30.045111131 +0000 UTC m=+1852.300011338" Jul 15 16:47:30.649484 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:30.649452 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:30.654103 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:30.654082 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:31.028937 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:31.028842 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:31.029832 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:31.029814 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:47.038505 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.038422 2560 patch_prober.go:28] interesting pod/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 16:47:47.038505 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.038480 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 16:47:47.120737 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.120709 2560 generic.go:356] "Generic (PLEG): container finished" podID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerID="4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769" exitCode=0 Jul 15 16:47:47.120840 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.120788 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" event={"ID":"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d","Type":"ContainerDied","Data":"4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769"} Jul 15 16:47:47.121220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.121205 2560 scope.go:117] "RemoveContainer" containerID="4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769" Jul 15 16:47:47.122272 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.122245 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:47:47.122403 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.122311 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:47:47.122403 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:47.122345 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:47:48.128406 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:48.128377 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" event={"ID":"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d","Type":"ContainerStarted","Data":"7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38"} Jul 15 16:47:48.649193 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:48.649165 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:48.649367 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:48.649274 2560 patch_prober.go:28] interesting pod/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 16:47:48.649367 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:48.649319 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 16:47:49.649138 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:49.649103 2560 patch_prober.go:28] interesting pod/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 16:47:49.649540 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:49.649166 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 16:47:50.649263 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:50.649224 2560 patch_prober.go:28] interesting pod/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 16:47:50.649741 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:50.649297 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 16:47:51.648926 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:51.648894 2560 patch_prober.go:28] interesting pod/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 15 16:47:51.649097 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:51.648946 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 15 16:47:52.652397 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:52.652366 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:52.652850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:52.652753 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:47:52.653456 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:47:52.653434 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:49:00.223452 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:00.223419 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5"] Jul 15 16:49:00.223998 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:00.223620 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" containerID="cri-o://7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38" gracePeriod=30 Jul 15 16:49:05.468633 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.468609 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:49:05.520696 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520616 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-credential-socket\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520696 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520664 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmqkt\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-kube-api-access-rmqkt\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520699 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-token\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520760 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-certs\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520802 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-socket\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520836 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-envoy\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520858 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-data\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.520932 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.520918 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:49:05.521251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521016 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istiod-ca-cert\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.521251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521052 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-podinfo\") pod \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\" (UID: \"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d\") " Jul 15 16:49:05.521251 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521164 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:49:05.521406 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521343 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:49:05.521406 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521396 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-credential-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.521508 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521412 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.521752 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:49:05.521988 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.521959 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:49:05.523603 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.523567 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-kube-api-access-rmqkt" (OuterVolumeSpecName: "kube-api-access-rmqkt") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "kube-api-access-rmqkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:49:05.523829 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.523787 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:49:05.523934 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.523844 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:49:05.524067 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.524040 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" (UID: "9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 16:49:05.545933 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.545903 2560 generic.go:356] "Generic (PLEG): container finished" podID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerID="7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38" exitCode=0 Jul 15 16:49:05.546109 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.545946 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" event={"ID":"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d","Type":"ContainerDied","Data":"7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38"} Jul 15 16:49:05.546109 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.545969 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" Jul 15 16:49:05.546109 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.545994 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5" event={"ID":"9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d","Type":"ContainerDied","Data":"37291e6da9e2ca36d34abffa64b66f06bd83ede2ffb46ce5cb5a744994db52e1"} Jul 15 16:49:05.546109 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.546017 2560 scope.go:117] "RemoveContainer" containerID="7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38" Jul 15 16:49:05.555930 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.555911 2560 scope.go:117] "RemoveContainer" containerID="4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769" Jul 15 16:49:05.564461 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.564439 2560 scope.go:117] "RemoveContainer" containerID="7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38" Jul 15 16:49:05.564743 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:49:05.564724 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38\": container with ID starting with 7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38 not found: ID does not exist" containerID="7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38" Jul 15 16:49:05.564803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.564751 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38"} err="failed to get container status \"7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38\": rpc error: code = NotFound desc = could not find container \"7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38\": container with ID starting with 7833d6872ec717e4b2709aab3ec5c1ed888427b16b5c4f899df2e1d8e5537b38 not found: ID does not exist" Jul 15 16:49:05.564803 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.564770 2560 scope.go:117] "RemoveContainer" containerID="4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769" Jul 15 16:49:05.565088 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:49:05.565062 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769\": container with ID starting with 4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769 not found: ID does not exist" containerID="4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769" Jul 15 16:49:05.565154 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.565094 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769"} err="failed to get container status \"4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769\": rpc error: code = NotFound desc = could not find container \"4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769\": container with ID starting with 4e7cd0fe0beca3f43039a066d5bad06d26b429063bcef3d7551395e664fe8769 not found: ID does not exist" Jul 15 16:49:05.573329 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.573307 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5"] Jul 15 16:49:05.578313 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.578287 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-77b034d1-openshift-default-7567dff7f5-njph5"] Jul 15 16:49:05.622022 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622000 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-workload-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.622022 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622021 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-envoy\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.622149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622029 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-data\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.622149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622037 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istiod-ca-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.622149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622045 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-podinfo\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.622149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622053 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqkt\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-kube-api-access-rmqkt\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:05.622149 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:05.622061 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d-istio-token\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:49:06.312760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:49:06.312725 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" path="/var/lib/kubelet/pods/9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d/volumes" Jul 15 16:51:38.408139 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:38.408110 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:51:38.415573 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:38.415552 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:51:54.122622 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.122590 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr"] Jul 15 16:51:54.123015 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.122976 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" Jul 15 16:51:54.123015 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.122990 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" Jul 15 16:51:54.123107 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.123073 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" Jul 15 16:51:54.123107 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.123083 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e66905-5639-4556-bdf6-fb5e1cb08407" containerName="istio-proxy" Jul 15 16:51:54.126136 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.126117 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.130458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.130438 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dc2354a8\"/\"openshift-service-ca.crt\"" Jul 15 16:51:54.130598 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.130440 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dc2354a8\"/\"default-dockercfg-trdcd\"" Jul 15 16:51:54.130598 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.130440 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dc2354a8\"/\"kube-root-ca.crt\"" Jul 15 16:51:54.130598 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.130487 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dc2354a8\"/\"e2e-lifecycle-model-dc2354a8-kserve-self-signed-certs\"" Jul 15 16:51:54.138280 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.138259 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr"] Jul 15 16:51:54.226763 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.226732 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-home\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.226961 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.226771 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-model-cache\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.226961 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.226790 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-tmp-dir\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.226961 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.226895 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqt4\" (UniqueName: \"kubernetes.io/projected/1403fa4e-9590-4761-8b76-67ac9ef4da79-kube-api-access-7wqt4\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.227081 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.226972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1403fa4e-9590-4761-8b76-67ac9ef4da79-tls-certs\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.227081 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.227009 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-dshm\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327403 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-home\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-model-cache\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327453 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-tmp-dir\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqt4\" (UniqueName: \"kubernetes.io/projected/1403fa4e-9590-4761-8b76-67ac9ef4da79-kube-api-access-7wqt4\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327739 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1403fa4e-9590-4761-8b76-67ac9ef4da79-tls-certs\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327739 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-dshm\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.327923 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327898 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-home\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.328004 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327954 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-tmp-dir\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.328004 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.327973 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-model-cache\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.329892 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.329854 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-dshm\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.330078 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.330061 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1403fa4e-9590-4761-8b76-67ac9ef4da79-tls-certs\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.338038 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.338005 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqt4\" (UniqueName: \"kubernetes.io/projected/1403fa4e-9590-4761-8b76-67ac9ef4da79-kube-api-access-7wqt4\") pod \"e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.436805 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.436730 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:51:54.569400 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:54.569368 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr"] Jul 15 16:51:54.569903 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:51:54.569864 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1403fa4e_9590_4761_8b76_67ac9ef4da79.slice/crio-a5d87f111f9b26b6cb3d5c544a5e0daa7f7c3f5c9b6f9446edaa99fe4337982c WatchSource:0}: Error finding container a5d87f111f9b26b6cb3d5c544a5e0daa7f7c3f5c9b6f9446edaa99fe4337982c: Status 404 returned error can't find the container with id a5d87f111f9b26b6cb3d5c544a5e0daa7f7c3f5c9b6f9446edaa99fe4337982c Jul 15 16:51:55.460411 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:55.460376 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" event={"ID":"1403fa4e-9590-4761-8b76-67ac9ef4da79","Type":"ContainerStarted","Data":"cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914"} Jul 15 16:51:55.460411 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:55.460417 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" event={"ID":"1403fa4e-9590-4761-8b76-67ac9ef4da79","Type":"ContainerStarted","Data":"a5d87f111f9b26b6cb3d5c544a5e0daa7f7c3f5c9b6f9446edaa99fe4337982c"} Jul 15 16:51:55.482488 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:51:55.482438 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" podStartSLOduration=1.482423252 podStartE2EDuration="1.482423252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:51:55.479524723 +0000 UTC m=+2117.734424928" watchObservedRunningTime="2026-07-15 16:51:55.482423252 +0000 UTC m=+2117.737323503" Jul 15 16:52:04.437679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:04.437641 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:52:04.438113 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:04.437713 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:52:04.450116 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:04.450092 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:52:04.518850 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:04.518819 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:52:41.364102 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:41.364063 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr"] Jul 15 16:52:56.384577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.384510 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" podUID="1403fa4e-9590-4761-8b76-67ac9ef4da79" containerName="main" containerID="cri-o://cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914" gracePeriod=45 Jul 15 16:52:56.633349 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.633326 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:52:56.691781 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.691711 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wqt4\" (UniqueName: \"kubernetes.io/projected/1403fa4e-9590-4761-8b76-67ac9ef4da79-kube-api-access-7wqt4\") pod \"1403fa4e-9590-4761-8b76-67ac9ef4da79\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " Jul 15 16:52:56.691781 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.691748 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-dshm\") pod \"1403fa4e-9590-4761-8b76-67ac9ef4da79\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " Jul 15 16:52:56.691781 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.691781 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1403fa4e-9590-4761-8b76-67ac9ef4da79-tls-certs\") pod \"1403fa4e-9590-4761-8b76-67ac9ef4da79\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " Jul 15 16:52:56.692020 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.691811 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-tmp-dir\") pod \"1403fa4e-9590-4761-8b76-67ac9ef4da79\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " Jul 15 16:52:56.692020 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.691894 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-home\") pod \"1403fa4e-9590-4761-8b76-67ac9ef4da79\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " Jul 15 16:52:56.692020 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.691952 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-model-cache\") pod \"1403fa4e-9590-4761-8b76-67ac9ef4da79\" (UID: \"1403fa4e-9590-4761-8b76-67ac9ef4da79\") " Jul 15 16:52:56.692180 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.692141 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1403fa4e-9590-4761-8b76-67ac9ef4da79" (UID: "1403fa4e-9590-4761-8b76-67ac9ef4da79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:52:56.692180 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.692160 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-home" (OuterVolumeSpecName: "home") pod "1403fa4e-9590-4761-8b76-67ac9ef4da79" (UID: "1403fa4e-9590-4761-8b76-67ac9ef4da79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:52:56.692307 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.692289 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-tmp-dir\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:52:56.692372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.692307 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-home\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:52:56.692372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.692335 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-model-cache" (OuterVolumeSpecName: "model-cache") pod "1403fa4e-9590-4761-8b76-67ac9ef4da79" (UID: "1403fa4e-9590-4761-8b76-67ac9ef4da79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:52:56.694028 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.693999 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1403fa4e-9590-4761-8b76-67ac9ef4da79-kube-api-access-7wqt4" (OuterVolumeSpecName: "kube-api-access-7wqt4") pod "1403fa4e-9590-4761-8b76-67ac9ef4da79" (UID: "1403fa4e-9590-4761-8b76-67ac9ef4da79"). InnerVolumeSpecName "kube-api-access-7wqt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:52:56.694243 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.694227 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1403fa4e-9590-4761-8b76-67ac9ef4da79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1403fa4e-9590-4761-8b76-67ac9ef4da79" (UID: "1403fa4e-9590-4761-8b76-67ac9ef4da79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:52:56.694243 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.694234 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-dshm" (OuterVolumeSpecName: "dshm") pod "1403fa4e-9590-4761-8b76-67ac9ef4da79" (UID: "1403fa4e-9590-4761-8b76-67ac9ef4da79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:52:56.793415 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.793383 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1403fa4e-9590-4761-8b76-67ac9ef4da79-tls-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:52:56.793415 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.793409 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-model-cache\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:52:56.793415 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.793420 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqt4\" (UniqueName: \"kubernetes.io/projected/1403fa4e-9590-4761-8b76-67ac9ef4da79-kube-api-access-7wqt4\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:52:56.793686 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.793434 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1403fa4e-9590-4761-8b76-67ac9ef4da79-dshm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:52:56.794433 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.794409 2560 generic.go:356] "Generic (PLEG): container finished" podID="1403fa4e-9590-4761-8b76-67ac9ef4da79" containerID="cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914" exitCode=0 Jul 15 16:52:56.794500 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.794456 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" Jul 15 16:52:56.794538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.794492 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" event={"ID":"1403fa4e-9590-4761-8b76-67ac9ef4da79","Type":"ContainerDied","Data":"cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914"} Jul 15 16:52:56.794578 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.794540 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr" event={"ID":"1403fa4e-9590-4761-8b76-67ac9ef4da79","Type":"ContainerDied","Data":"a5d87f111f9b26b6cb3d5c544a5e0daa7f7c3f5c9b6f9446edaa99fe4337982c"} Jul 15 16:52:56.794578 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.794564 2560 scope.go:117] "RemoveContainer" containerID="cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914" Jul 15 16:52:56.804060 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.804025 2560 scope.go:117] "RemoveContainer" containerID="cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914" Jul 15 16:52:56.804289 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:52:56.804271 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914\": container with ID starting with cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914 not found: ID does not exist" containerID="cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914" Jul 15 16:52:56.804342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.804299 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914"} err="failed to get container status \"cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914\": rpc error: code = NotFound desc = could not find container \"cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914\": container with ID starting with cebc158b01b9d4347bd4d05f30615613341dcecf582fc6f25284badeb0f39914 not found: ID does not exist" Jul 15 16:52:56.823329 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.823309 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr"] Jul 15 16:52:56.838112 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:56.838092 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dc2354a8/e2e-lifecycle-model-dc2354a8-kserve-6b6dc94df9-g6krr"] Jul 15 16:52:58.311134 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:52:58.311104 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1403fa4e-9590-4761-8b76-67ac9ef4da79" path="/var/lib/kubelet/pods/1403fa4e-9590-4761-8b76-67ac9ef4da79/volumes" Jul 15 16:54:43.171864 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.171826 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j"] Jul 15 16:54:43.172454 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.172422 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" Jul 15 16:54:43.172500 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.172459 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" Jul 15 16:54:43.172500 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.172472 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1403fa4e-9590-4761-8b76-67ac9ef4da79" containerName="main" Jul 15 16:54:43.172500 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.172481 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1403fa4e-9590-4761-8b76-67ac9ef4da79" containerName="main" Jul 15 16:54:43.172616 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.172604 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1403fa4e-9590-4761-8b76-67ac9ef4da79" containerName="main" Jul 15 16:54:43.172654 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.172621 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9918a6bd-b92f-4b1b-8f0c-87b04fe1de7d" containerName="istio-proxy" Jul 15 16:54:43.176038 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.176017 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.178911 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.178897 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cd9127-openshift-default-dockercfg-ngv59\"" Jul 15 16:54:43.187377 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.187356 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j"] Jul 15 16:54:43.208730 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2knt\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-kube-api-access-w2knt\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208836 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-socket\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208836 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208764 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-data\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208836 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208806 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istiod-ca-cert\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208971 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208844 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-credential-socket\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208971 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208895 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-certs\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208971 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208925 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-podinfo\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.208971 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.208942 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-token\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.209096 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.209017 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-envoy\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309568 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309539 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-certs\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-podinfo\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-token\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-envoy\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309664 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2knt\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-kube-api-access-w2knt\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309683 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-socket\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-data\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.309760 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istiod-ca-cert\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.310162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.309897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-credential-socket\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.310162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.310022 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-certs\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.310162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.310059 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-socket\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.310162 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.310119 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-data\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.310370 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.310175 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-credential-socket\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.310591 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.310574 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istiod-ca-cert\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.311986 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.311956 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-envoy\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.312312 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.312292 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-podinfo\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.318544 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.318512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2knt\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-kube-api-access-w2knt\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.318658 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.318641 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-token\") pod \"e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.488538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.488452 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:43.624099 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.624064 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j"] Jul 15 16:54:43.626504 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:54:43.626322 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6b0097_03d9_45b0_ab43_f2025c2b0a84.slice/crio-0522d576f62aed9fa19037636e8c4340adb8370294f67ec9ea58ce3ada9559d1 WatchSource:0}: Error finding container 0522d576f62aed9fa19037636e8c4340adb8370294f67ec9ea58ce3ada9559d1: Status 404 returned error can't find the container with id 0522d576f62aed9fa19037636e8c4340adb8370294f67ec9ea58ce3ada9559d1 Jul 15 16:54:43.629715 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.629684 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:54:43.629796 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.629760 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:54:43.629840 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:43.629799 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:54:44.383228 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:44.383184 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" event={"ID":"7a6b0097-03d9-45b0-ab43-f2025c2b0a84","Type":"ContainerStarted","Data":"e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48"} Jul 15 16:54:44.383596 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:44.383249 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" event={"ID":"7a6b0097-03d9-45b0-ab43-f2025c2b0a84","Type":"ContainerStarted","Data":"0522d576f62aed9fa19037636e8c4340adb8370294f67ec9ea58ce3ada9559d1"} Jul 15 16:54:44.489305 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:44.489277 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:44.494444 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:44.494424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:44.528661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:44.528618 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podStartSLOduration=1.528600067 podStartE2EDuration="1.528600067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:54:44.439792473 +0000 UTC m=+2286.694692679" watchObservedRunningTime="2026-07-15 16:54:44.528600067 +0000 UTC m=+2286.783500274" Jul 15 16:54:45.388290 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:45.388255 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:54:45.389250 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:54:45.389232 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:55:00.389716 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.389634 2560 patch_prober.go:28] interesting pod/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 16:55:00.389716 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.389694 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 16:55:00.474763 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.474728 2560 generic.go:356] "Generic (PLEG): container finished" podID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerID="e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48" exitCode=0 Jul 15 16:55:00.474953 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.474804 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" event={"ID":"7a6b0097-03d9-45b0-ab43-f2025c2b0a84","Type":"ContainerDied","Data":"e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48"} Jul 15 16:55:00.475301 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.475285 2560 scope.go:117] "RemoveContainer" containerID="e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48" Jul 15 16:55:00.476365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.476298 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:55:00.476474 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.476379 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:55:00.476474 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:00.476426 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:55:01.482616 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:01.482585 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" event={"ID":"7a6b0097-03d9-45b0-ab43-f2025c2b0a84","Type":"ContainerStarted","Data":"65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa"} Jul 15 16:55:01.488793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:01.488775 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:55:01.489024 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:01.489005 2560 patch_prober.go:28] interesting pod/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 16:55:01.489069 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:01.489048 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 16:55:02.488956 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:02.488923 2560 patch_prober.go:28] interesting pod/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 16:55:02.489435 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:02.488977 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 16:55:03.489194 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:03.489162 2560 patch_prober.go:28] interesting pod/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 16:55:03.489577 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:03.489219 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 16:55:04.489538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:04.489494 2560 patch_prober.go:28] interesting pod/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 15 16:55:04.489963 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:04.489572 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 15 16:55:05.492806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:05.492775 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:55:05.493287 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:05.493265 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:55:05.493964 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:05.493947 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:55:15.627473 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.627438 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv"] Jul 15 16:55:15.630793 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.630776 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.635172 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.635146 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cd9127\"/\"openshift-service-ca.crt\"" Jul 15 16:55:15.635330 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.635183 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cd9127\"/\"default-dockercfg-wg9xv\"" Jul 15 16:55:15.635330 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.635290 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cd9127\"/\"kube-root-ca.crt\"" Jul 15 16:55:15.635676 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.635659 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cd9127\"/\"test-model-cd9127-kserve-self-signed-certs\"" Jul 15 16:55:15.642507 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.642484 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv"] Jul 15 16:55:15.732262 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.732225 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-home\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.732435 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.732265 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5bn\" (UniqueName: \"kubernetes.io/projected/70a4ae82-42fe-4274-8171-4a0c3e46883a-kube-api-access-tb5bn\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.732435 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.732335 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a4ae82-42fe-4274-8171-4a0c3e46883a-tls-certs\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.732435 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.732360 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-dshm\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.732435 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.732427 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-model-cache\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.732614 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.732451 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-tmp-dir\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.832963 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.832934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-home\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.832963 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.832968 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5bn\" (UniqueName: \"kubernetes.io/projected/70a4ae82-42fe-4274-8171-4a0c3e46883a-kube-api-access-tb5bn\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a4ae82-42fe-4274-8171-4a0c3e46883a-tls-certs\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833052 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-dshm\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833133 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-model-cache\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-tmp-dir\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833395 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833377 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-home\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833490 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833469 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-model-cache\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.833540 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.833512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-tmp-dir\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.835447 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.835425 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-dshm\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.835652 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.835636 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a4ae82-42fe-4274-8171-4a0c3e46883a-tls-certs\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.841363 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.841339 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5bn\" (UniqueName: \"kubernetes.io/projected/70a4ae82-42fe-4274-8171-4a0c3e46883a-kube-api-access-tb5bn\") pod \"test-model-cd9127-kserve-847fb8c589-8cbhv\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:15.941274 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:15.941202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:16.074290 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:16.074264 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv"] Jul 15 16:55:16.076163 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:55:16.076135 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70a4ae82_42fe_4274_8171_4a0c3e46883a.slice/crio-d2f05d60ddc94dffe7dc0a1f23d1bb7c250113cb469ce794cc10ba1a2439efd2 WatchSource:0}: Error finding container d2f05d60ddc94dffe7dc0a1f23d1bb7c250113cb469ce794cc10ba1a2439efd2: Status 404 returned error can't find the container with id d2f05d60ddc94dffe7dc0a1f23d1bb7c250113cb469ce794cc10ba1a2439efd2 Jul 15 16:55:16.569847 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:16.569804 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" event={"ID":"70a4ae82-42fe-4274-8171-4a0c3e46883a","Type":"ContainerStarted","Data":"9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276"} Jul 15 16:55:16.570045 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:16.569855 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" event={"ID":"70a4ae82-42fe-4274-8171-4a0c3e46883a","Type":"ContainerStarted","Data":"d2f05d60ddc94dffe7dc0a1f23d1bb7c250113cb469ce794cc10ba1a2439efd2"} Jul 15 16:55:16.590972 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:16.590925 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" podStartSLOduration=1.590912278 podStartE2EDuration="1.590912278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:55:16.589174163 +0000 UTC m=+2318.844074385" watchObservedRunningTime="2026-07-15 16:55:16.590912278 +0000 UTC m=+2318.845812485" Jul 15 16:55:25.942113 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:25.942071 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:25.942506 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:25.942130 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:25.954883 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:25.954851 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:26.636347 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:26.636315 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:41.325661 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:41.325568 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv"] Jul 15 16:55:56.346071 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.346028 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" podUID="70a4ae82-42fe-4274-8171-4a0c3e46883a" containerName="main" containerID="cri-o://9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276" gracePeriod=45 Jul 15 16:55:56.589795 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.589772 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:56.602058 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602001 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-dshm\") pod \"70a4ae82-42fe-4274-8171-4a0c3e46883a\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " Jul 15 16:55:56.602058 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602037 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-tmp-dir\") pod \"70a4ae82-42fe-4274-8171-4a0c3e46883a\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " Jul 15 16:55:56.602226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602066 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-home\") pod \"70a4ae82-42fe-4274-8171-4a0c3e46883a\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " Jul 15 16:55:56.602226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602086 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb5bn\" (UniqueName: \"kubernetes.io/projected/70a4ae82-42fe-4274-8171-4a0c3e46883a-kube-api-access-tb5bn\") pod \"70a4ae82-42fe-4274-8171-4a0c3e46883a\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " Jul 15 16:55:56.602226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602131 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-model-cache\") pod \"70a4ae82-42fe-4274-8171-4a0c3e46883a\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " Jul 15 16:55:56.602226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602169 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a4ae82-42fe-4274-8171-4a0c3e46883a-tls-certs\") pod \"70a4ae82-42fe-4274-8171-4a0c3e46883a\" (UID: \"70a4ae82-42fe-4274-8171-4a0c3e46883a\") " Jul 15 16:55:56.602434 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602293 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-home" (OuterVolumeSpecName: "home") pod "70a4ae82-42fe-4274-8171-4a0c3e46883a" (UID: "70a4ae82-42fe-4274-8171-4a0c3e46883a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:55:56.602434 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602308 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70a4ae82-42fe-4274-8171-4a0c3e46883a" (UID: "70a4ae82-42fe-4274-8171-4a0c3e46883a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:55:56.602434 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602394 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-model-cache" (OuterVolumeSpecName: "model-cache") pod "70a4ae82-42fe-4274-8171-4a0c3e46883a" (UID: "70a4ae82-42fe-4274-8171-4a0c3e46883a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:55:56.602588 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602530 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-tmp-dir\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:55:56.602588 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602542 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-home\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:55:56.602588 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.602551 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-model-cache\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:55:56.604210 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.604190 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-dshm" (OuterVolumeSpecName: "dshm") pod "70a4ae82-42fe-4274-8171-4a0c3e46883a" (UID: "70a4ae82-42fe-4274-8171-4a0c3e46883a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:55:56.604289 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.604240 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a4ae82-42fe-4274-8171-4a0c3e46883a-kube-api-access-tb5bn" (OuterVolumeSpecName: "kube-api-access-tb5bn") pod "70a4ae82-42fe-4274-8171-4a0c3e46883a" (UID: "70a4ae82-42fe-4274-8171-4a0c3e46883a"). InnerVolumeSpecName "kube-api-access-tb5bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:55:56.604326 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.604302 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a4ae82-42fe-4274-8171-4a0c3e46883a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70a4ae82-42fe-4274-8171-4a0c3e46883a" (UID: "70a4ae82-42fe-4274-8171-4a0c3e46883a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 16:55:56.703607 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.703573 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a4ae82-42fe-4274-8171-4a0c3e46883a-tls-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:55:56.703607 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.703601 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70a4ae82-42fe-4274-8171-4a0c3e46883a-dshm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:55:56.703808 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.703636 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5bn\" (UniqueName: \"kubernetes.io/projected/70a4ae82-42fe-4274-8171-4a0c3e46883a-kube-api-access-tb5bn\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:55:56.792080 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.792046 2560 generic.go:356] "Generic (PLEG): container finished" podID="70a4ae82-42fe-4274-8171-4a0c3e46883a" containerID="9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276" exitCode=0 Jul 15 16:55:56.792255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.792092 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" Jul 15 16:55:56.792255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.792109 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" event={"ID":"70a4ae82-42fe-4274-8171-4a0c3e46883a","Type":"ContainerDied","Data":"9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276"} Jul 15 16:55:56.792255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.792143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv" event={"ID":"70a4ae82-42fe-4274-8171-4a0c3e46883a","Type":"ContainerDied","Data":"d2f05d60ddc94dffe7dc0a1f23d1bb7c250113cb469ce794cc10ba1a2439efd2"} Jul 15 16:55:56.792255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.792159 2560 scope.go:117] "RemoveContainer" containerID="9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276" Jul 15 16:55:56.801712 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.801694 2560 scope.go:117] "RemoveContainer" containerID="9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276" Jul 15 16:55:56.801999 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:55:56.801981 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276\": container with ID starting with 9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276 not found: ID does not exist" containerID="9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276" Jul 15 16:55:56.802045 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.802008 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276"} err="failed to get container status \"9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276\": rpc error: code = NotFound desc = could not find container \"9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276\": container with ID starting with 9195c159fb717247d31db3f44ac4917752fa11650e4bec1fa544ec34581d6276 not found: ID does not exist" Jul 15 16:55:56.814950 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.814927 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv"] Jul 15 16:55:56.819311 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:56.819285 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cd9127/test-model-cd9127-kserve-847fb8c589-8cbhv"] Jul 15 16:55:58.310918 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:55:58.310862 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a4ae82-42fe-4274-8171-4a0c3e46883a" path="/var/lib/kubelet/pods/70a4ae82-42fe-4274-8171-4a0c3e46883a/volumes" Jul 15 16:56:33.466195 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:33.466160 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j"] Jul 15 16:56:33.466778 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:33.466441 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" containerID="cri-o://65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa" gracePeriod=30 Jul 15 16:56:35.495844 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:35.495802 2560 patch_prober.go:28] interesting pod/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 16:56:35.496230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:35.495890 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 16:56:38.445282 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.445255 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:56:38.458186 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.458159 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 16:56:38.720765 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.720743 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:56:38.802984 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.802932 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istiod-ca-cert\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.802984 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.802995 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-podinfo\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803030 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-token\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803083 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-credential-socket\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803110 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-certs\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803140 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-socket\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803189 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-envoy\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803215 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2knt\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-kube-api-access-w2knt\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803514 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803243 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-data\") pod \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\" (UID: \"7a6b0097-03d9-45b0-ab43-f2025c2b0a84\") " Jul 15 16:56:38.803514 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803388 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:56:38.803514 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803439 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:56:38.803679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803523 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:56:38.803679 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803577 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:56:38.803786 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803718 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istiod-ca-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.803786 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803736 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-credential-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.803786 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803749 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.803786 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.803762 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-workload-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.804150 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.804128 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:56:38.805455 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.805424 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 16:56:38.805455 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.805436 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:56:38.805845 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.805829 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:56:38.805933 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.805900 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-kube-api-access-w2knt" (OuterVolumeSpecName: "kube-api-access-w2knt") pod "7a6b0097-03d9-45b0-ab43-f2025c2b0a84" (UID: "7a6b0097-03d9-45b0-ab43-f2025c2b0a84"). InnerVolumeSpecName "kube-api-access-w2knt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:56:38.904537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.904499 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-data\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.904537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.904530 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-podinfo\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.904537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.904543 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-token\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.904537 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.904550 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-istio-envoy\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:38.904813 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:38.904558 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2knt\" (UniqueName: \"kubernetes.io/projected/7a6b0097-03d9-45b0-ab43-f2025c2b0a84-kube-api-access-w2knt\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:56:39.021414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.021321 2560 generic.go:356] "Generic (PLEG): container finished" podID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerID="65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa" exitCode=0 Jul 15 16:56:39.021414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.021357 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" event={"ID":"7a6b0097-03d9-45b0-ab43-f2025c2b0a84","Type":"ContainerDied","Data":"65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa"} Jul 15 16:56:39.021414 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.021389 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" Jul 15 16:56:39.021671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.021441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j" event={"ID":"7a6b0097-03d9-45b0-ab43-f2025c2b0a84","Type":"ContainerDied","Data":"0522d576f62aed9fa19037636e8c4340adb8370294f67ec9ea58ce3ada9559d1"} Jul 15 16:56:39.021671 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.021465 2560 scope.go:117] "RemoveContainer" containerID="65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa" Jul 15 16:56:39.031688 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.031667 2560 scope.go:117] "RemoveContainer" containerID="e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48" Jul 15 16:56:39.039894 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.039854 2560 scope.go:117] "RemoveContainer" containerID="65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa" Jul 15 16:56:39.040153 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:56:39.040136 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa\": container with ID starting with 65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa not found: ID does not exist" containerID="65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa" Jul 15 16:56:39.040203 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.040162 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa"} err="failed to get container status \"65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa\": rpc error: code = NotFound desc = could not find container \"65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa\": container with ID starting with 65a681de8b515fb38499617bdd41f3bf85b8efba172396c3bbc19ef55dd996aa not found: ID does not exist" Jul 15 16:56:39.040203 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.040179 2560 scope.go:117] "RemoveContainer" containerID="e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48" Jul 15 16:56:39.040405 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:56:39.040387 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48\": container with ID starting with e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48 not found: ID does not exist" containerID="e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48" Jul 15 16:56:39.040446 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.040413 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48"} err="failed to get container status \"e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48\": rpc error: code = NotFound desc = could not find container \"e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48\": container with ID starting with e9b9bfedc5d720187a92b6d9dfc62419140153e70bea2eacc1607d176516cb48 not found: ID does not exist" Jul 15 16:56:39.044983 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.044956 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j"] Jul 15 16:56:39.049702 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:39.049677 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cd9127-openshift-default-77c8579b55-9695j"] Jul 15 16:56:40.311209 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:40.311179 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" path="/var/lib/kubelet/pods/7a6b0097-03d9-45b0-ab43-f2025c2b0a84/volumes" Jul 15 16:56:42.888017 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.887985 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw"] Jul 15 16:56:42.888405 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.888395 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" Jul 15 16:56:42.888458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.888409 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" Jul 15 16:56:42.888458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.888423 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4ae82-42fe-4274-8171-4a0c3e46883a" containerName="main" Jul 15 16:56:42.888458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.888429 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4ae82-42fe-4274-8171-4a0c3e46883a" containerName="main" Jul 15 16:56:42.888565 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.888489 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" Jul 15 16:56:42.888565 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.888497 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a4ae82-42fe-4274-8171-4a0c3e46883a" containerName="main" Jul 15 16:56:42.893173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.893153 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.895847 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.895828 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-90f61e-openshift-default-dockercfg-ddp7g\"" Jul 15 16:56:42.904290 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.904266 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw"] Jul 15 16:56:42.942196 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-token\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942335 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-data\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942335 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942232 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-podinfo\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942335 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942250 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5r5\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-kube-api-access-wx5r5\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942335 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36431a60-1915-422d-a90a-ec95d92c7f5a-istiod-ca-cert\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942545 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-socket\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942545 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-certs\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942545 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942527 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-credential-socket\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:42.942655 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:42.942596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-envoy\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044031 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.043996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-certs\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044196 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044043 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-credential-socket\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044196 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044158 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-envoy\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044317 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044214 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-token\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044317 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-data\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044317 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044300 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-podinfo\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044463 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044319 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5r5\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-kube-api-access-wx5r5\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044463 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36431a60-1915-422d-a90a-ec95d92c7f5a-istiod-ca-cert\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044463 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-socket\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044463 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044362 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-credential-socket\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044463 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044391 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-certs\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044756 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-data\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.044821 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.044802 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-socket\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.045040 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.045021 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36431a60-1915-422d-a90a-ec95d92c7f5a-istiod-ca-cert\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.046458 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.046438 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-envoy\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.046764 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.046747 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-podinfo\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.052424 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.052400 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5r5\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-kube-api-access-wx5r5\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.052506 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.052479 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-token\") pod \"e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.203465 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.203384 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:43.348804 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.348781 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw"] Jul 15 16:56:43.350546 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:56:43.350518 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36431a60_1915_422d_a90a_ec95d92c7f5a.slice/crio-c6e6a33618a7c1a7ddb8775259376c3c5dfa23ce1017ff3ca2104b8d57ca4366 WatchSource:0}: Error finding container c6e6a33618a7c1a7ddb8775259376c3c5dfa23ce1017ff3ca2104b8d57ca4366: Status 404 returned error can't find the container with id c6e6a33618a7c1a7ddb8775259376c3c5dfa23ce1017ff3ca2104b8d57ca4366 Jul 15 16:56:43.352706 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.352674 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:56:43.352810 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.352740 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:56:43.352810 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:43.352772 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:56:44.051951 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:44.051919 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" event={"ID":"36431a60-1915-422d-a90a-ec95d92c7f5a","Type":"ContainerStarted","Data":"eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74"} Jul 15 16:56:44.052306 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:44.051981 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" event={"ID":"36431a60-1915-422d-a90a-ec95d92c7f5a","Type":"ContainerStarted","Data":"c6e6a33618a7c1a7ddb8775259376c3c5dfa23ce1017ff3ca2104b8d57ca4366"} Jul 15 16:56:44.076303 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:44.076251 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podStartSLOduration=2.076233646 podStartE2EDuration="2.076233646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:56:44.073836848 +0000 UTC m=+2406.328737079" watchObservedRunningTime="2026-07-15 16:56:44.076233646 +0000 UTC m=+2406.331133852" Jul 15 16:56:44.204475 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:44.204442 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:44.209132 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:44.209110 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:45.057471 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:45.057435 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:56:45.058395 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:56:45.058375 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:01.099039 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:01.098996 2560 patch_prober.go:28] interesting pod/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 16:57:01.099519 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:01.099061 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 16:57:02.158571 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:02.158531 2560 generic.go:356] "Generic (PLEG): container finished" podID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerID="eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74" exitCode=0 Jul 15 16:57:02.158995 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:02.158630 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" event={"ID":"36431a60-1915-422d-a90a-ec95d92c7f5a","Type":"ContainerDied","Data":"eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74"} Jul 15 16:57:02.159113 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:02.159098 2560 scope.go:117] "RemoveContainer" containerID="eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74" Jul 15 16:57:02.160197 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:02.160169 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:57:02.160275 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:02.160237 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:57:02.160275 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:02.160267 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:57:03.166197 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:03.166158 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" event={"ID":"36431a60-1915-422d-a90a-ec95d92c7f5a","Type":"ContainerStarted","Data":"ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b"} Jul 15 16:57:03.204313 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:03.204270 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:03.204517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:03.204401 2560 patch_prober.go:28] interesting pod/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 15 16:57:03.204517 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:03.204479 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 15 16:57:04.204425 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:04.204386 2560 patch_prober.go:28] interesting pod/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 15 16:57:04.205070 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:04.205037 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 15 16:57:05.204828 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:05.204789 2560 patch_prober.go:28] interesting pod/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 15 16:57:05.205459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:05.205423 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 15 16:57:06.203861 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:06.203825 2560 patch_prober.go:28] interesting pod/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 15 16:57:06.204230 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:06.204198 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 15 16:57:07.207471 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:07.207438 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:07.207863 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:07.207705 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:07.208386 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:07.208370 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:51.017608 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:51.017573 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw"] Jul 15 16:57:51.018112 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:51.017786 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" containerID="cri-o://ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b" gracePeriod=30 Jul 15 16:57:52.210370 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:52.210326 2560 patch_prober.go:28] interesting pod/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 16:57:52.210820 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:52.210402 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 16:57:56.269127 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.269104 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:56.338201 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338182 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-certs\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338225 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx5r5\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-kube-api-access-wx5r5\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338271 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36431a60-1915-422d-a90a-ec95d92c7f5a-istiod-ca-cert\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338295 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-data\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338310 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-token\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338325 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-podinfo\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338640 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338395 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-socket\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338640 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338431 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-envoy\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338640 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338488 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-credential-socket\") pod \"36431a60-1915-422d-a90a-ec95d92c7f5a\" (UID: \"36431a60-1915-422d-a90a-ec95d92c7f5a\") " Jul 15 16:57:56.338640 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338498 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:57:56.338924 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338659 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36431a60-1915-422d-a90a-ec95d92c7f5a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 16:57:56.338924 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338665 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:57:56.338924 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.338860 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:57:56.339118 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.339058 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.339196 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.339173 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-workload-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.339272 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.339199 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36431a60-1915-422d-a90a-ec95d92c7f5a-istiod-ca-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.339272 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.339220 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-data" (OuterVolumeSpecName: "istio-data") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:57:56.341175 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.341147 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 16:57:56.341175 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.341155 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:57:56.341342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.341176 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-kube-api-access-wx5r5" (OuterVolumeSpecName: "kube-api-access-wx5r5") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "kube-api-access-wx5r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 16:57:56.341385 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.341352 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36431a60-1915-422d-a90a-ec95d92c7f5a" (UID: "36431a60-1915-422d-a90a-ec95d92c7f5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 16:57:56.440091 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.440051 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-credential-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.440091 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.440082 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx5r5\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-kube-api-access-wx5r5\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.440091 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.440093 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-data\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.440091 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.440103 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-token\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.440369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.440110 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-podinfo\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.440369 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.440119 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36431a60-1915-422d-a90a-ec95d92c7f5a-istio-envoy\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 16:57:56.454245 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.454216 2560 generic.go:356] "Generic (PLEG): container finished" podID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerID="ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b" exitCode=0 Jul 15 16:57:56.454360 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.454262 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" Jul 15 16:57:56.454360 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.454312 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" event={"ID":"36431a60-1915-422d-a90a-ec95d92c7f5a","Type":"ContainerDied","Data":"ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b"} Jul 15 16:57:56.454436 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.454367 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw" event={"ID":"36431a60-1915-422d-a90a-ec95d92c7f5a","Type":"ContainerDied","Data":"c6e6a33618a7c1a7ddb8775259376c3c5dfa23ce1017ff3ca2104b8d57ca4366"} Jul 15 16:57:56.454436 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.454390 2560 scope.go:117] "RemoveContainer" containerID="ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b" Jul 15 16:57:56.464018 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.464002 2560 scope.go:117] "RemoveContainer" containerID="eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74" Jul 15 16:57:56.472427 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.472410 2560 scope.go:117] "RemoveContainer" containerID="ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b" Jul 15 16:57:56.472678 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:57:56.472659 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b\": container with ID starting with ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b not found: ID does not exist" containerID="ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b" Jul 15 16:57:56.472724 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.472689 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b"} err="failed to get container status \"ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b\": rpc error: code = NotFound desc = could not find container \"ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b\": container with ID starting with ebc11422d162f6cb0670ce8a1ef84c7e23f6db37b424f3161d00d144a4f4d33b not found: ID does not exist" Jul 15 16:57:56.472724 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.472707 2560 scope.go:117] "RemoveContainer" containerID="eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74" Jul 15 16:57:56.472968 ip-10-0-136-51 kubenswrapper[2560]: E0715 16:57:56.472950 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74\": container with ID starting with eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74 not found: ID does not exist" containerID="eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74" Jul 15 16:57:56.473031 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.472979 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74"} err="failed to get container status \"eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74\": rpc error: code = NotFound desc = could not find container \"eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74\": container with ID starting with eec5b25434ef53374292c461deac0942b08a095cdf2ee188fb3fdc61f6174d74 not found: ID does not exist" Jul 15 16:57:56.515675 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.515644 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw"] Jul 15 16:57:56.521461 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:56.521435 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90f61e-openshift-default-66568fd56c-q8zjw"] Jul 15 16:57:58.311249 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:57:58.311214 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" path="/var/lib/kubelet/pods/36431a60-1915-422d-a90a-ec95d92c7f5a/volumes" Jul 15 16:58:00.268828 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.268801 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz"] Jul 15 16:58:00.269226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.269191 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" Jul 15 16:58:00.269226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.269203 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" Jul 15 16:58:00.269226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.269221 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" Jul 15 16:58:00.269226 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.269227 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" Jul 15 16:58:00.269378 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.269294 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" Jul 15 16:58:00.269378 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.269305 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6b0097-03d9-45b0-ab43-f2025c2b0a84" containerName="istio-proxy" Jul 15 16:58:00.273580 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.273563 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.276761 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.276739 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-021cd5-openshift-default-dockercfg-cxngw\"" Jul 15 16:58:00.285090 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.285071 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz"] Jul 15 16:58:00.378126 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378089 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnm9\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-kube-api-access-bgnm9\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378126 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-credential-socket\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378223 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istiod-ca-cert\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378254 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-data\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378293 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-podinfo\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378323 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-token\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378353 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-socket\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378386 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-certs\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.378459 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.378425 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-envoy\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479329 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-token\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-socket\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-certs\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-envoy\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479474 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnm9\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-kube-api-access-bgnm9\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479538 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479507 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-credential-socket\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479820 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479547 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istiod-ca-cert\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479820 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479571 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-data\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.479820 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-podinfo\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.480011 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479855 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-socket\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.480011 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.479908 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-data\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.480094 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.480082 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-credential-socket\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.480271 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.480244 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-certs\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.480342 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.480303 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istiod-ca-cert\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.481902 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.481863 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-envoy\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.482200 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.482184 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-podinfo\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.488465 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.488436 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-token\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.488571 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.488553 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnm9\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-kube-api-access-bgnm9\") pod \"e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.585073 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.584989 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:00.722822 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.722786 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz"] Jul 15 16:58:00.724077 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:58:00.724051 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a0a30a_8c0d_44b2_8e7d_6ce34c0799c0.slice/crio-dfa762a427e80b939591d99f12491f166950bddaf1f08a58b353c3957e817b69 WatchSource:0}: Error finding container dfa762a427e80b939591d99f12491f166950bddaf1f08a58b353c3957e817b69: Status 404 returned error can't find the container with id dfa762a427e80b939591d99f12491f166950bddaf1f08a58b353c3957e817b69 Jul 15 16:58:00.726220 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.726190 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:58:00.726318 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.726255 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:58:00.726318 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:00.726285 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:58:01.488947 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:01.488913 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" event={"ID":"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0","Type":"ContainerStarted","Data":"1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407"} Jul 15 16:58:01.489394 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:01.488970 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" event={"ID":"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0","Type":"ContainerStarted","Data":"dfa762a427e80b939591d99f12491f166950bddaf1f08a58b353c3957e817b69"} Jul 15 16:58:01.512933 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:01.512864 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podStartSLOduration=1.512848513 podStartE2EDuration="1.512848513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:58:01.512181993 +0000 UTC m=+2483.767082202" watchObservedRunningTime="2026-07-15 16:58:01.512848513 +0000 UTC m=+2483.767748719" Jul 15 16:58:01.585641 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:01.585606 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:01.591173 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:01.591151 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:02.494553 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:02.494521 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:02.495467 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:02.495449 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:18.537067 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:18.537010 2560 patch_prober.go:28] interesting pod/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 16:58:18.537607 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:18.537095 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 16:58:19.584959 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:19.584924 2560 generic.go:356] "Generic (PLEG): container finished" podID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerID="1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407" exitCode=0 Jul 15 16:58:19.585372 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:19.584997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" event={"ID":"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0","Type":"ContainerDied","Data":"1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407"} Jul 15 16:58:19.585421 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:19.585398 2560 scope.go:117] "RemoveContainer" containerID="1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407" Jul 15 16:58:19.586381 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:19.586353 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:58:19.586470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:19.586415 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:58:19.586470 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:19.586446 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 16:58:20.592618 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:20.592584 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" event={"ID":"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0","Type":"ContainerStarted","Data":"c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2"} Jul 15 16:58:21.586166 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:21.586134 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:21.586358 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:21.586337 2560 patch_prober.go:28] interesting pod/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 16:58:21.586409 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:21.586384 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 16:58:22.585609 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:22.585572 2560 patch_prober.go:28] interesting pod/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 16:58:22.586044 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:22.585629 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 16:58:23.586388 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:23.586344 2560 patch_prober.go:28] interesting pod/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 16:58:23.586902 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:23.586430 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 16:58:24.585801 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:24.585762 2560 patch_prober.go:28] interesting pod/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 15 16:58:24.586012 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:24.585822 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 15 16:58:25.588784 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:25.588754 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:25.589255 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:25.589132 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:25.589900 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:25.589866 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 16:58:48.769962 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.769928 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db"] Jul 15 16:58:48.770359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.770331 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" Jul 15 16:58:48.770359 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.770341 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" Jul 15 16:58:48.770438 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.770402 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="36431a60-1915-422d-a90a-ec95d92c7f5a" containerName="istio-proxy" Jul 15 16:58:48.772525 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.772509 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.776027 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.776008 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"kube-root-ca.crt\"" Jul 15 16:58:48.777088 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.777070 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"auth-test-model-021cd5-kserve-self-signed-certs\"" Jul 15 16:58:48.777160 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.777137 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"default-dockercfg-7tmlc\"" Jul 15 16:58:48.779672 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.779654 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"openshift-service-ca.crt\"" Jul 15 16:58:48.788044 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.788024 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db"] Jul 15 16:58:48.847686 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.847656 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-home\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.847686 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.847689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vncj\" (UniqueName: \"kubernetes.io/projected/cb931a83-723a-410f-8704-b1a8dc07ebf3-kube-api-access-8vncj\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.847858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.847735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb931a83-723a-410f-8704-b1a8dc07ebf3-tls-certs\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.847858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.847775 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-tmp-dir\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.847858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.847830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-dshm\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.847858 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.847849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-model-cache\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949017 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-dshm\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949055 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-model-cache\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949122 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-home\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949148 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vncj\" (UniqueName: \"kubernetes.io/projected/cb931a83-723a-410f-8704-b1a8dc07ebf3-kube-api-access-8vncj\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949321 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949289 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb931a83-723a-410f-8704-b1a8dc07ebf3-tls-certs\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949497 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-tmp-dir\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949558 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949522 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-model-cache\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949614 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949561 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-home\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.949738 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.949719 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-tmp-dir\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.951283 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.951260 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-dshm\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.951571 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.951555 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb931a83-723a-410f-8704-b1a8dc07ebf3-tls-certs\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:48.958763 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:48.958741 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vncj\" (UniqueName: \"kubernetes.io/projected/cb931a83-723a-410f-8704-b1a8dc07ebf3-kube-api-access-8vncj\") pod \"auth-test-model-021cd5-kserve-599564f847-tx8db\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:49.081913 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:49.081797 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:49.225971 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:49.225943 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db"] Jul 15 16:58:49.226784 ip-10-0-136-51 kubenswrapper[2560]: W0715 16:58:49.226764 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb931a83_723a_410f_8704_b1a8dc07ebf3.slice/crio-e38db5a979cb972d958ca11b35b17ea8f411593909ad73aa57f0ce9310299778 WatchSource:0}: Error finding container e38db5a979cb972d958ca11b35b17ea8f411593909ad73aa57f0ce9310299778: Status 404 returned error can't find the container with id e38db5a979cb972d958ca11b35b17ea8f411593909ad73aa57f0ce9310299778 Jul 15 16:58:49.768812 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:49.768771 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" event={"ID":"cb931a83-723a-410f-8704-b1a8dc07ebf3","Type":"ContainerStarted","Data":"e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa"} Jul 15 16:58:49.768812 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:49.768811 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" event={"ID":"cb931a83-723a-410f-8704-b1a8dc07ebf3","Type":"ContainerStarted","Data":"e38db5a979cb972d958ca11b35b17ea8f411593909ad73aa57f0ce9310299778"} Jul 15 16:58:49.793806 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:49.793750 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" podStartSLOduration=1.793734242 podStartE2EDuration="1.793734242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 16:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 16:58:49.791409186 +0000 UTC m=+2532.046309393" watchObservedRunningTime="2026-07-15 16:58:49.793734242 +0000 UTC m=+2532.048634449" Jul 15 16:58:59.082912 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:59.082863 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:59.083348 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:59.082928 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:59.095365 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:59.095342 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 16:58:59.841657 ip-10-0-136-51 kubenswrapper[2560]: I0715 16:58:59.841631 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 17:00:43.190168 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:43.190127 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db"] Jul 15 17:00:58.210513 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.210472 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" podUID="cb931a83-723a-410f-8704-b1a8dc07ebf3" containerName="main" containerID="cri-o://e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa" gracePeriod=45 Jul 15 17:00:58.458971 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.458886 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 17:00:58.475621 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.475554 2560 generic.go:356] "Generic (PLEG): container finished" podID="cb931a83-723a-410f-8704-b1a8dc07ebf3" containerID="e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa" exitCode=0 Jul 15 17:00:58.475621 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.475595 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" Jul 15 17:00:58.475796 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.475635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" event={"ID":"cb931a83-723a-410f-8704-b1a8dc07ebf3","Type":"ContainerDied","Data":"e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa"} Jul 15 17:00:58.475796 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.475677 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db" event={"ID":"cb931a83-723a-410f-8704-b1a8dc07ebf3","Type":"ContainerDied","Data":"e38db5a979cb972d958ca11b35b17ea8f411593909ad73aa57f0ce9310299778"} Jul 15 17:00:58.475796 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.475693 2560 scope.go:117] "RemoveContainer" containerID="e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa" Jul 15 17:00:58.487898 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.487861 2560 scope.go:117] "RemoveContainer" containerID="e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa" Jul 15 17:00:58.488189 ip-10-0-136-51 kubenswrapper[2560]: E0715 17:00:58.488170 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa\": container with ID starting with e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa not found: ID does not exist" containerID="e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa" Jul 15 17:00:58.488255 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.488202 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa"} err="failed to get container status \"e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa\": rpc error: code = NotFound desc = could not find container \"e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa\": container with ID starting with e7e2187ffc79b4a53252b5ffe2be05aadb0f7e489a3690b0d19d0002574641aa not found: ID does not exist" Jul 15 17:00:58.548559 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.548534 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vncj\" (UniqueName: \"kubernetes.io/projected/cb931a83-723a-410f-8704-b1a8dc07ebf3-kube-api-access-8vncj\") pod \"cb931a83-723a-410f-8704-b1a8dc07ebf3\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " Jul 15 17:00:58.548666 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.548572 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb931a83-723a-410f-8704-b1a8dc07ebf3-tls-certs\") pod \"cb931a83-723a-410f-8704-b1a8dc07ebf3\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " Jul 15 17:00:58.548666 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.548601 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-tmp-dir\") pod \"cb931a83-723a-410f-8704-b1a8dc07ebf3\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " Jul 15 17:00:58.548666 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.548629 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-home\") pod \"cb931a83-723a-410f-8704-b1a8dc07ebf3\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " Jul 15 17:00:58.548666 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.548659 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-model-cache\") pod \"cb931a83-723a-410f-8704-b1a8dc07ebf3\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " Jul 15 17:00:58.548919 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.548688 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-dshm\") pod \"cb931a83-723a-410f-8704-b1a8dc07ebf3\" (UID: \"cb931a83-723a-410f-8704-b1a8dc07ebf3\") " Jul 15 17:00:58.549052 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.549010 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb931a83-723a-410f-8704-b1a8dc07ebf3" (UID: "cb931a83-723a-410f-8704-b1a8dc07ebf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:00:58.549052 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.549025 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb931a83-723a-410f-8704-b1a8dc07ebf3" (UID: "cb931a83-723a-410f-8704-b1a8dc07ebf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:00:58.549305 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.549107 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-home" (OuterVolumeSpecName: "home") pod "cb931a83-723a-410f-8704-b1a8dc07ebf3" (UID: "cb931a83-723a-410f-8704-b1a8dc07ebf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:00:58.551039 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.551018 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-dshm" (OuterVolumeSpecName: "dshm") pod "cb931a83-723a-410f-8704-b1a8dc07ebf3" (UID: "cb931a83-723a-410f-8704-b1a8dc07ebf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:00:58.551176 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.551148 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb931a83-723a-410f-8704-b1a8dc07ebf3-kube-api-access-8vncj" (OuterVolumeSpecName: "kube-api-access-8vncj") pod "cb931a83-723a-410f-8704-b1a8dc07ebf3" (UID: "cb931a83-723a-410f-8704-b1a8dc07ebf3"). InnerVolumeSpecName "kube-api-access-8vncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 17:00:58.551250 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.551199 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb931a83-723a-410f-8704-b1a8dc07ebf3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb931a83-723a-410f-8704-b1a8dc07ebf3" (UID: "cb931a83-723a-410f-8704-b1a8dc07ebf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 17:00:58.653200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.650770 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-tmp-dir\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:00:58.653200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.650801 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-home\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:00:58.653200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.650823 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-model-cache\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:00:58.653200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.650838 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb931a83-723a-410f-8704-b1a8dc07ebf3-dshm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:00:58.653200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.650851 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vncj\" (UniqueName: \"kubernetes.io/projected/cb931a83-723a-410f-8704-b1a8dc07ebf3-kube-api-access-8vncj\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:00:58.653200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.650864 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb931a83-723a-410f-8704-b1a8dc07ebf3-tls-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:00:58.799366 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.799333 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db"] Jul 15 17:00:58.801586 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:00:58.801566 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/auth-test-model-021cd5-kserve-599564f847-tx8db"] Jul 15 17:01:00.310852 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:00.310818 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb931a83-723a-410f-8704-b1a8dc07ebf3" path="/var/lib/kubelet/pods/cb931a83-723a-410f-8704-b1a8dc07ebf3/volumes" Jul 15 17:01:17.842625 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.842595 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2"] Jul 15 17:01:17.843037 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.843004 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb931a83-723a-410f-8704-b1a8dc07ebf3" containerName="main" Jul 15 17:01:17.843037 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.843016 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb931a83-723a-410f-8704-b1a8dc07ebf3" containerName="main" Jul 15 17:01:17.843119 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.843087 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb931a83-723a-410f-8704-b1a8dc07ebf3" containerName="main" Jul 15 17:01:17.845256 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.845239 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:17.848026 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.848004 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"openshift-service-ca.crt\"" Jul 15 17:01:17.849099 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.849079 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"kube-root-ca.crt\"" Jul 15 17:01:17.849207 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.849168 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"sub-test-model-021cd5-kserve-self-signed-certs\"" Jul 15 17:01:17.849395 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.849378 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"default-dockercfg-7tmlc\"" Jul 15 17:01:17.857214 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.857191 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2"] Jul 15 17:01:17.921249 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.921214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-dshm\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:17.921385 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.921282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-model-cache\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:17.921385 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.921318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-home\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:17.921512 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.921390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1bd465e-ee68-4a59-b511-7730b18d96e1-tls-certs\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:17.921512 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.921421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69np9\" (UniqueName: \"kubernetes.io/projected/b1bd465e-ee68-4a59-b511-7730b18d96e1-kube-api-access-69np9\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:17.921512 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:17.921504 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-tmp-dir\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.022511 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022486 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-tmp-dir\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.022651 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022531 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-dshm\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.022651 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022565 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-model-cache\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.022651 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-home\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.022651 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022607 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1bd465e-ee68-4a59-b511-7730b18d96e1-tls-certs\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.022651 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022630 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69np9\" (UniqueName: \"kubernetes.io/projected/b1bd465e-ee68-4a59-b511-7730b18d96e1-kube-api-access-69np9\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.023009 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.022980 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-tmp-dir\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.023112 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.023012 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-model-cache\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.023112 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.023061 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-home\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.024762 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.024744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-dshm\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.025068 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.025051 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1bd465e-ee68-4a59-b511-7730b18d96e1-tls-certs\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.030651 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.030630 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69np9\" (UniqueName: \"kubernetes.io/projected/b1bd465e-ee68-4a59-b511-7730b18d96e1-kube-api-access-69np9\") pod \"sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.155086 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.155001 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:18.289382 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.289354 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2"] Jul 15 17:01:18.290698 ip-10-0-136-51 kubenswrapper[2560]: W0715 17:01:18.290675 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1bd465e_ee68_4a59_b511_7730b18d96e1.slice/crio-eaf6537717986e2c735d34205c6d7fe83d6592d3580776406e758448b3461ad3 WatchSource:0}: Error finding container eaf6537717986e2c735d34205c6d7fe83d6592d3580776406e758448b3461ad3: Status 404 returned error can't find the container with id eaf6537717986e2c735d34205c6d7fe83d6592d3580776406e758448b3461ad3 Jul 15 17:01:18.584044 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.583959 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" event={"ID":"b1bd465e-ee68-4a59-b511-7730b18d96e1","Type":"ContainerStarted","Data":"124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb"} Jul 15 17:01:18.584044 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.584001 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" event={"ID":"b1bd465e-ee68-4a59-b511-7730b18d96e1","Type":"ContainerStarted","Data":"eaf6537717986e2c735d34205c6d7fe83d6592d3580776406e758448b3461ad3"} Jul 15 17:01:18.605014 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:18.604964 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" podStartSLOduration=1.604949123 podStartE2EDuration="1.604949123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 17:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 17:01:18.602082705 +0000 UTC m=+2680.856982911" watchObservedRunningTime="2026-07-15 17:01:18.604949123 +0000 UTC m=+2680.859849329" Jul 15 17:01:28.156054 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:28.156016 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:28.156452 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:28.156068 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:28.169107 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:28.169086 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:28.656839 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:28.656813 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:01:38.491460 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:38.491377 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 17:01:38.503583 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:38.503560 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-65mm5_5912f599-2ac6-4169-900f-e5576f6d6e67/ovn-acl-logging/0.log" Jul 15 17:01:53.236268 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:01:53.236234 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2"] Jul 15 17:02:08.255768 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.255707 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" podUID="b1bd465e-ee68-4a59-b511-7730b18d96e1" containerName="main" containerID="cri-o://124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb" gracePeriod=45 Jul 15 17:02:08.500067 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.500036 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:02:08.570678 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570608 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-home\") pod \"b1bd465e-ee68-4a59-b511-7730b18d96e1\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " Jul 15 17:02:08.570678 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570656 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69np9\" (UniqueName: \"kubernetes.io/projected/b1bd465e-ee68-4a59-b511-7730b18d96e1-kube-api-access-69np9\") pod \"b1bd465e-ee68-4a59-b511-7730b18d96e1\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " Jul 15 17:02:08.570678 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570674 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-dshm\") pod \"b1bd465e-ee68-4a59-b511-7730b18d96e1\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " Jul 15 17:02:08.570981 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570721 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-model-cache\") pod \"b1bd465e-ee68-4a59-b511-7730b18d96e1\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " Jul 15 17:02:08.570981 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570779 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1bd465e-ee68-4a59-b511-7730b18d96e1-tls-certs\") pod \"b1bd465e-ee68-4a59-b511-7730b18d96e1\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " Jul 15 17:02:08.570981 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570814 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-home" (OuterVolumeSpecName: "home") pod "b1bd465e-ee68-4a59-b511-7730b18d96e1" (UID: "b1bd465e-ee68-4a59-b511-7730b18d96e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:02:08.570981 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.570829 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-tmp-dir\") pod \"b1bd465e-ee68-4a59-b511-7730b18d96e1\" (UID: \"b1bd465e-ee68-4a59-b511-7730b18d96e1\") " Jul 15 17:02:08.571201 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.571052 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1bd465e-ee68-4a59-b511-7730b18d96e1" (UID: "b1bd465e-ee68-4a59-b511-7730b18d96e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:02:08.571201 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.571126 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1bd465e-ee68-4a59-b511-7730b18d96e1" (UID: "b1bd465e-ee68-4a59-b511-7730b18d96e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:02:08.571201 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.571175 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-model-cache\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:02:08.571201 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.571193 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-home\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:02:08.572807 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.572779 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-dshm" (OuterVolumeSpecName: "dshm") pod "b1bd465e-ee68-4a59-b511-7730b18d96e1" (UID: "b1bd465e-ee68-4a59-b511-7730b18d96e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:02:08.572807 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.572796 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bd465e-ee68-4a59-b511-7730b18d96e1-kube-api-access-69np9" (OuterVolumeSpecName: "kube-api-access-69np9") pod "b1bd465e-ee68-4a59-b511-7730b18d96e1" (UID: "b1bd465e-ee68-4a59-b511-7730b18d96e1"). InnerVolumeSpecName "kube-api-access-69np9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 17:02:08.572807 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.572797 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bd465e-ee68-4a59-b511-7730b18d96e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1bd465e-ee68-4a59-b511-7730b18d96e1" (UID: "b1bd465e-ee68-4a59-b511-7730b18d96e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 17:02:08.671889 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.671846 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1bd465e-ee68-4a59-b511-7730b18d96e1-tls-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:02:08.671889 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.671881 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-tmp-dir\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:02:08.671889 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.671890 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69np9\" (UniqueName: \"kubernetes.io/projected/b1bd465e-ee68-4a59-b511-7730b18d96e1-kube-api-access-69np9\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:02:08.672072 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.671899 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1bd465e-ee68-4a59-b511-7730b18d96e1-dshm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:02:08.866096 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.866066 2560 generic.go:356] "Generic (PLEG): container finished" podID="b1bd465e-ee68-4a59-b511-7730b18d96e1" containerID="124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb" exitCode=0 Jul 15 17:02:08.866232 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.866114 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" event={"ID":"b1bd465e-ee68-4a59-b511-7730b18d96e1","Type":"ContainerDied","Data":"124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb"} Jul 15 17:02:08.866232 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.866147 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" event={"ID":"b1bd465e-ee68-4a59-b511-7730b18d96e1","Type":"ContainerDied","Data":"eaf6537717986e2c735d34205c6d7fe83d6592d3580776406e758448b3461ad3"} Jul 15 17:02:08.866232 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.866165 2560 scope.go:117] "RemoveContainer" containerID="124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb" Jul 15 17:02:08.866232 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.866117 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2" Jul 15 17:02:08.876168 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.876150 2560 scope.go:117] "RemoveContainer" containerID="124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb" Jul 15 17:02:08.876443 ip-10-0-136-51 kubenswrapper[2560]: E0715 17:02:08.876411 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb\": container with ID starting with 124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb not found: ID does not exist" containerID="124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb" Jul 15 17:02:08.876515 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.876451 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb"} err="failed to get container status \"124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb\": rpc error: code = NotFound desc = could not find container \"124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb\": container with ID starting with 124edfbffdae0c2a61fb844d7f51ce2af6736998572ffd41951888bc86596ffb not found: ID does not exist" Jul 15 17:02:08.890167 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.890147 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2"] Jul 15 17:02:08.894517 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:08.894498 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/sub-test-model-021cd5-kserve-64f8dfb64f-hgbm2"] Jul 15 17:02:10.310497 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:10.310456 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bd465e-ee68-4a59-b511-7730b18d96e1" path="/var/lib/kubelet/pods/b1bd465e-ee68-4a59-b511-7730b18d96e1/volumes" Jul 15 17:02:46.535545 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.535504 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh"] Jul 15 17:02:46.536052 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.535899 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bd465e-ee68-4a59-b511-7730b18d96e1" containerName="main" Jul 15 17:02:46.536052 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.535910 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bd465e-ee68-4a59-b511-7730b18d96e1" containerName="main" Jul 15 17:02:46.536052 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.535981 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bd465e-ee68-4a59-b511-7730b18d96e1" containerName="main" Jul 15 17:02:46.538173 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.538157 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.541110 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.541071 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"kube-root-ca.crt\"" Jul 15 17:02:46.541110 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.541073 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"openshift-service-ca.crt\"" Jul 15 17:02:46.542278 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.542258 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"default-dockercfg-7tmlc\"" Jul 15 17:02:46.542388 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.542278 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-021cd5\"/\"rate-test-model-021cd5-kserve-self-signed-certs\"" Jul 15 17:02:46.547986 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.547965 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh"] Jul 15 17:02:46.612816 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.612788 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-dshm\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.612972 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.612821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tls-certs\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.612972 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.612847 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-home\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.612972 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.612890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7p2\" (UniqueName: \"kubernetes.io/projected/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-kube-api-access-ms7p2\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.613088 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.612967 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-model-cache\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.613088 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.613028 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tmp-dir\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.713575 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713534 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-dshm\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.713575 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713574 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tls-certs\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.713951 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713596 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-home\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.713951 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713726 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7p2\" (UniqueName: \"kubernetes.io/projected/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-kube-api-access-ms7p2\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.713951 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713807 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-model-cache\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.713951 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713908 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tmp-dir\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.714202 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.713969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-home\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.714202 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.714166 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-model-cache\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.714202 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.714195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tmp-dir\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.715926 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.715904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-dshm\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.716365 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.716345 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tls-certs\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.728621 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.728588 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7p2\" (UniqueName: \"kubernetes.io/projected/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-kube-api-access-ms7p2\") pod \"rate-test-model-021cd5-kserve-8685f4f479-tn7dh\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.848376 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.848343 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:46.988596 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:46.988570 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh"] Jul 15 17:02:46.990610 ip-10-0-136-51 kubenswrapper[2560]: W0715 17:02:46.990581 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f994d3_efe1_43fd_8d3f_f8762d7b9a79.slice/crio-0d5e232dbf1620ea958281d6441bdae242b8584eb47f66c6f452edb335fdbda0 WatchSource:0}: Error finding container 0d5e232dbf1620ea958281d6441bdae242b8584eb47f66c6f452edb335fdbda0: Status 404 returned error can't find the container with id 0d5e232dbf1620ea958281d6441bdae242b8584eb47f66c6f452edb335fdbda0 Jul 15 17:02:47.074488 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:47.074442 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" event={"ID":"89f994d3-efe1-43fd-8d3f-f8762d7b9a79","Type":"ContainerStarted","Data":"d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f"} Jul 15 17:02:47.074605 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:47.074504 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" event={"ID":"89f994d3-efe1-43fd-8d3f-f8762d7b9a79","Type":"ContainerStarted","Data":"0d5e232dbf1620ea958281d6441bdae242b8584eb47f66c6f452edb335fdbda0"} Jul 15 17:02:47.101322 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:47.101238 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" podStartSLOduration=1.101222563 podStartE2EDuration="1.101222563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 17:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 17:02:47.099034168 +0000 UTC m=+2769.353934404" watchObservedRunningTime="2026-07-15 17:02:47.101222563 +0000 UTC m=+2769.356122768" Jul 15 17:02:56.849474 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:56.849430 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:56.849941 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:56.849491 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:56.862035 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:56.862013 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:02:57.147184 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:02:57.147101 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:03:30.614455 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:30.614420 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh"] Jul 15 17:03:45.635123 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.635081 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" podUID="89f994d3-efe1-43fd-8d3f-f8762d7b9a79" containerName="main" containerID="cri-o://d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f" gracePeriod=45 Jul 15 17:03:45.880038 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.880007 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:03:45.996319 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996241 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tmp-dir\") pod \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " Jul 15 17:03:45.996319 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7p2\" (UniqueName: \"kubernetes.io/projected/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-kube-api-access-ms7p2\") pod \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " Jul 15 17:03:45.996319 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996318 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-dshm\") pod \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " Jul 15 17:03:45.996602 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996343 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tls-certs\") pod \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " Jul 15 17:03:45.996602 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996364 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-home\") pod \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " Jul 15 17:03:45.996602 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996406 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-model-cache\") pod \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\" (UID: \"89f994d3-efe1-43fd-8d3f-f8762d7b9a79\") " Jul 15 17:03:45.996602 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996532 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89f994d3-efe1-43fd-8d3f-f8762d7b9a79" (UID: "89f994d3-efe1-43fd-8d3f-f8762d7b9a79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:03:45.996822 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996704 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tmp-dir\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:45.996822 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996695 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-model-cache" (OuterVolumeSpecName: "model-cache") pod "89f994d3-efe1-43fd-8d3f-f8762d7b9a79" (UID: "89f994d3-efe1-43fd-8d3f-f8762d7b9a79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:03:45.996822 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.996715 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-home" (OuterVolumeSpecName: "home") pod "89f994d3-efe1-43fd-8d3f-f8762d7b9a79" (UID: "89f994d3-efe1-43fd-8d3f-f8762d7b9a79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:03:45.998444 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.998422 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-dshm" (OuterVolumeSpecName: "dshm") pod "89f994d3-efe1-43fd-8d3f-f8762d7b9a79" (UID: "89f994d3-efe1-43fd-8d3f-f8762d7b9a79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:03:45.998550 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.998456 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-kube-api-access-ms7p2" (OuterVolumeSpecName: "kube-api-access-ms7p2") pod "89f994d3-efe1-43fd-8d3f-f8762d7b9a79" (UID: "89f994d3-efe1-43fd-8d3f-f8762d7b9a79"). InnerVolumeSpecName "kube-api-access-ms7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 17:03:45.998550 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:45.998477 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89f994d3-efe1-43fd-8d3f-f8762d7b9a79" (UID: "89f994d3-efe1-43fd-8d3f-f8762d7b9a79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 17:03:46.104012 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.103973 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-model-cache\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:46.104192 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.104005 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7p2\" (UniqueName: \"kubernetes.io/projected/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-kube-api-access-ms7p2\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:46.104192 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.104082 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-dshm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:46.104192 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.104090 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-tls-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:46.104192 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.104098 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f994d3-efe1-43fd-8d3f-f8762d7b9a79-home\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:46.399016 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.398980 2560 generic.go:356] "Generic (PLEG): container finished" podID="89f994d3-efe1-43fd-8d3f-f8762d7b9a79" containerID="d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f" exitCode=0 Jul 15 17:03:46.399183 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.399040 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" Jul 15 17:03:46.399183 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.399067 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" event={"ID":"89f994d3-efe1-43fd-8d3f-f8762d7b9a79","Type":"ContainerDied","Data":"d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f"} Jul 15 17:03:46.399183 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.399116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh" event={"ID":"89f994d3-efe1-43fd-8d3f-f8762d7b9a79","Type":"ContainerDied","Data":"0d5e232dbf1620ea958281d6441bdae242b8584eb47f66c6f452edb335fdbda0"} Jul 15 17:03:46.399183 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.399139 2560 scope.go:117] "RemoveContainer" containerID="d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f" Jul 15 17:03:46.408361 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.408341 2560 scope.go:117] "RemoveContainer" containerID="d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f" Jul 15 17:03:46.408615 ip-10-0-136-51 kubenswrapper[2560]: E0715 17:03:46.408596 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f\": container with ID starting with d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f not found: ID does not exist" containerID="d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f" Jul 15 17:03:46.408663 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.408625 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f"} err="failed to get container status \"d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f\": rpc error: code = NotFound desc = could not find container \"d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f\": container with ID starting with d79e2ff1abed7819ea33a43252028d3aca73dfaa11126f839601e2327256df9f not found: ID does not exist" Jul 15 17:03:46.418515 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.418486 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh"] Jul 15 17:03:46.423106 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:46.423083 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-021cd5/rate-test-model-021cd5-kserve-8685f4f479-tn7dh"] Jul 15 17:03:47.357755 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.357720 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7410-139944546657312"] Jul 15 17:03:47.358172 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.358123 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f994d3-efe1-43fd-8d3f-f8762d7b9a79" containerName="main" Jul 15 17:03:47.358172 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.358135 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f994d3-efe1-43fd-8d3f-f8762d7b9a79" containerName="main" Jul 15 17:03:47.358243 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.358200 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f994d3-efe1-43fd-8d3f-f8762d7b9a79" containerName="main" Jul 15 17:03:47.361099 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.361083 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:47.364847 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.364826 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cmnsb\"" Jul 15 17:03:47.380903 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.380864 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7410-139944546657312"] Jul 15 17:03:47.415173 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.415144 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kzm\" (UniqueName: \"kubernetes.io/projected/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c-kube-api-access-d9kzm\") pod \"test-curl-7410-139944546657312\" (UID: \"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c\") " pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:47.516243 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.516203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kzm\" (UniqueName: \"kubernetes.io/projected/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c-kube-api-access-d9kzm\") pod \"test-curl-7410-139944546657312\" (UID: \"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c\") " pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:47.525354 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.525323 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kzm\" (UniqueName: \"kubernetes.io/projected/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c-kube-api-access-d9kzm\") pod \"test-curl-7410-139944546657312\" (UID: \"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c\") " pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:47.670152 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.670060 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:47.804616 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.804590 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7410-139944546657312"] Jul 15 17:03:47.807035 ip-10-0-136-51 kubenswrapper[2560]: W0715 17:03:47.807009 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf13d7124_ea5b_48cf_97c3_f3390e1f0b9c.slice/crio-e9e7dce20e6d001a3ecec00ed79bb237bea99ec78a32ad16d6ff2602961e4aa6 WatchSource:0}: Error finding container e9e7dce20e6d001a3ecec00ed79bb237bea99ec78a32ad16d6ff2602961e4aa6: Status 404 returned error can't find the container with id e9e7dce20e6d001a3ecec00ed79bb237bea99ec78a32ad16d6ff2602961e4aa6 Jul 15 17:03:47.808699 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:47.808680 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 17:03:48.312040 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:48.312004 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f994d3-efe1-43fd-8d3f-f8762d7b9a79" path="/var/lib/kubelet/pods/89f994d3-efe1-43fd-8d3f-f8762d7b9a79/volumes" Jul 15 17:03:48.415631 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:48.415594 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7410-139944546657312" event={"ID":"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c","Type":"ContainerStarted","Data":"e9e7dce20e6d001a3ecec00ed79bb237bea99ec78a32ad16d6ff2602961e4aa6"} Jul 15 17:03:49.422294 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:49.422255 2560 generic.go:356] "Generic (PLEG): container finished" podID="f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" containerID="e9ba3159990c56a17a64fdfa06558b07900af19d8cb386384496fc0f1de4ccaa" exitCode=0 Jul 15 17:03:49.422699 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:49.422345 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7410-139944546657312" event={"ID":"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c","Type":"ContainerDied","Data":"e9ba3159990c56a17a64fdfa06558b07900af19d8cb386384496fc0f1de4ccaa"} Jul 15 17:03:50.560647 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.560622 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:50.608420 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.608395 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7410-139944546657312_f13d7124-ea5b-48cf-97c3-f3390e1f0b9c/test-curl-7410-139944546657312/0.log" Jul 15 17:03:50.646727 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.646694 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9kzm\" (UniqueName: \"kubernetes.io/projected/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c-kube-api-access-d9kzm\") pod \"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c\" (UID: \"f13d7124-ea5b-48cf-97c3-f3390e1f0b9c\") " Jul 15 17:03:50.649238 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.649193 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c-kube-api-access-d9kzm" (OuterVolumeSpecName: "kube-api-access-d9kzm") pod "f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" (UID: "f13d7124-ea5b-48cf-97c3-f3390e1f0b9c"). InnerVolumeSpecName "kube-api-access-d9kzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 17:03:50.651599 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.651577 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7410-139944546657312"] Jul 15 17:03:50.656058 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.656032 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7410-139944546657312"] Jul 15 17:03:50.747683 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:50.747599 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kzm\" (UniqueName: \"kubernetes.io/projected/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c-kube-api-access-d9kzm\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:03:51.435936 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:51.435906 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e7dce20e6d001a3ecec00ed79bb237bea99ec78a32ad16d6ff2602961e4aa6" Jul 15 17:03:51.435936 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:51.435919 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:51.446487 ip-10-0-136-51 kubenswrapper[2560]: E0715 17:03:51.446458 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7410-139944546657312\" is forbidden: User \"system:node:ip-10-0-136-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-51.ec2.internal' and this object" podUID="f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" pod="opendatahub/test-curl-7410-139944546657312" Jul 15 17:03:52.309973 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:03:52.309896 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" path="/var/lib/kubelet/pods/f13d7124-ea5b-48cf-97c3-f3390e1f0b9c/volumes" Jul 15 17:04:42.937438 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:42.937353 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz"] Jul 15 17:04:42.938010 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:42.937572 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" containerID="cri-o://c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2" gracePeriod=30 Jul 15 17:04:43.656791 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:43.656760 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c78cc48c-2hz6l_6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe/authorino/0.log" Jul 15 17:04:48.196308 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.196287 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 17:04:48.354492 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354464 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgnm9\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-kube-api-access-bgnm9\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354626 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354496 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-token\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354626 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354534 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-credential-socket\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354626 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354551 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istiod-ca-cert\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354626 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354580 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-podinfo\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354626 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354613 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-socket\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354888 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354659 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-certs\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354888 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354713 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-data\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.354888 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354745 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-envoy\") pod \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\" (UID: \"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0\") " Jul 15 17:04:48.355054 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.354970 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:04:48.355111 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.355079 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-credential-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.355111 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.355084 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:04:48.355111 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.355092 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 17:04:48.355351 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.355324 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:04:48.356112 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.356088 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:04:48.356846 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.356822 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 17:04:48.356981 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.356891 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-kube-api-access-bgnm9" (OuterVolumeSpecName: "kube-api-access-bgnm9") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "kube-api-access-bgnm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 17:04:48.357157 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.357132 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 17:04:48.357198 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.357167 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" (UID: "51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 17:04:48.456134 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456110 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-certs\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456134 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456134 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-data\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456309 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456142 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-envoy\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456309 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456150 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnm9\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-kube-api-access-bgnm9\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456309 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456159 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-token\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456309 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456167 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istiod-ca-cert\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456309 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456174 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-istio-podinfo\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.456309 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.456182 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0-workload-socket\") on node \"ip-10-0-136-51.ec2.internal\" DevicePath \"\"" Jul 15 17:04:48.750483 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.750451 2560 generic.go:356] "Generic (PLEG): container finished" podID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerID="c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2" exitCode=0 Jul 15 17:04:48.750653 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.750506 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" event={"ID":"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0","Type":"ContainerDied","Data":"c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2"} Jul 15 17:04:48.750653 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.750512 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" Jul 15 17:04:48.750653 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.750548 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz" event={"ID":"51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0","Type":"ContainerDied","Data":"dfa762a427e80b939591d99f12491f166950bddaf1f08a58b353c3957e817b69"} Jul 15 17:04:48.750653 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.750571 2560 scope.go:117] "RemoveContainer" containerID="c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2" Jul 15 17:04:48.760966 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.760947 2560 scope.go:117] "RemoveContainer" containerID="1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407" Jul 15 17:04:48.769026 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.769010 2560 scope.go:117] "RemoveContainer" containerID="c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2" Jul 15 17:04:48.769277 ip-10-0-136-51 kubenswrapper[2560]: E0715 17:04:48.769258 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2\": container with ID starting with c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2 not found: ID does not exist" containerID="c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2" Jul 15 17:04:48.769348 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.769290 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2"} err="failed to get container status \"c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2\": rpc error: code = NotFound desc = could not find container \"c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2\": container with ID starting with c0c0d9b6d01f9193574baa27015032b56d384d02799d534d055c263e95e493f2 not found: ID does not exist" Jul 15 17:04:48.769348 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.769313 2560 scope.go:117] "RemoveContainer" containerID="1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407" Jul 15 17:04:48.769576 ip-10-0-136-51 kubenswrapper[2560]: E0715 17:04:48.769559 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407\": container with ID starting with 1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407 not found: ID does not exist" containerID="1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407" Jul 15 17:04:48.769617 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.769582 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407"} err="failed to get container status \"1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407\": rpc error: code = NotFound desc = could not find container \"1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407\": container with ID starting with 1655d7d0a013b974e0256fe8d29ba397206c83b7637cfffcc381d532f4cce407 not found: ID does not exist" Jul 15 17:04:48.773595 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.773574 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz"] Jul 15 17:04:48.777906 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:48.777861 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-021cd5-openshift-default-8b64cdd7f-hd8rz"] Jul 15 17:04:49.739345 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:49.739323 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lvlkg_a60d3362-5052-43eb-8f12-fc6d12f4e1db/server/0.log" Jul 15 17:04:49.860952 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:49.860926 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r7hzm_3d476544-32be-4eed-8dc5-5db817bc7dd4/manager/0.log" Jul 15 17:04:49.991127 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:49.991055 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599cdff4cb-2bp48_81199833-4935-4991-8d8f-7d86dce289ad/manager/0.log" Jul 15 17:04:49.997479 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:49.997462 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599cdff4cb-2bp48_81199833-4935-4991-8d8f-7d86dce289ad/copy-manifests/0.log" Jul 15 17:04:50.310879 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:50.310787 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" path="/var/lib/kubelet/pods/51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0/volumes" Jul 15 17:04:53.506416 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.506388 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r_c7ac79c2-4a48-4b7d-891d-6f706f2a6466/util/0.log" Jul 15 17:04:53.513486 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.513470 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r_c7ac79c2-4a48-4b7d-891d-6f706f2a6466/pull/0.log" Jul 15 17:04:53.519947 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.519926 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r_c7ac79c2-4a48-4b7d-891d-6f706f2a6466/extract/0.log" Jul 15 17:04:53.646113 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.646085 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw_1ab4cfc6-39e2-46e5-8161-feca59d8b6e0/util/0.log" Jul 15 17:04:53.652859 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.652844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw_1ab4cfc6-39e2-46e5-8161-feca59d8b6e0/pull/0.log" Jul 15 17:04:53.659797 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.659779 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw_1ab4cfc6-39e2-46e5-8161-feca59d8b6e0/extract/0.log" Jul 15 17:04:53.781626 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.781566 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx_0797ecaa-02dc-4f8a-a374-49ec62364a19/pull/0.log" Jul 15 17:04:53.788217 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.788194 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx_0797ecaa-02dc-4f8a-a374-49ec62364a19/extract/0.log" Jul 15 17:04:53.794303 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.794279 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx_0797ecaa-02dc-4f8a-a374-49ec62364a19/util/0.log" Jul 15 17:04:53.914023 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.913991 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884_3e350d22-7902-4628-9cae-469398433e27/util/0.log" Jul 15 17:04:53.920546 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.920528 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884_3e350d22-7902-4628-9cae-469398433e27/pull/0.log" Jul 15 17:04:53.927073 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:53.927052 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884_3e350d22-7902-4628-9cae-469398433e27/extract/0.log" Jul 15 17:04:54.057761 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:54.057671 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c78cc48c-2hz6l_6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe/authorino/0.log" Jul 15 17:04:54.428196 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:54.428169 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rvdb7_9b4b9e23-66e1-47f9-8145-db66dbebb217/kuadrant-console-plugin/0.log" Jul 15 17:04:55.341685 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:55.341642 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pp2zw_b172f886-2c3c-4d70-b50d-51f163e507d8/istio-proxy/0.log" Jul 15 17:04:55.596114 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:55.596016 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84484457c8-qx84h_b0bf833c-4532-4ce5-a10a-4d100a580207/kube-auth-proxy/0.log" Jul 15 17:04:56.586080 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:56.586044 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mrp54_d0d326ac-6b64-497d-82ca-9b8e98d8aa30/main/0.log" Jul 15 17:04:56.825653 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:56.825607 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nmxcf_29fba469-bd61-4909-83a5-d5c61f2f8f4f/main/0.log" Jul 15 17:04:57.201616 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:04:57.201590 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-n6b9j_f8fd0bef-97b2-484a-91c5-874b54f8e1aa/main/0.log" Jul 15 17:05:03.304886 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.304839 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4vtsf/must-gather-w5fvz"] Jul 15 17:05:03.305342 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305325 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" containerName="test-curl-7410-139944546657312" Jul 15 17:05:03.305394 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305344 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" containerName="test-curl-7410-139944546657312" Jul 15 17:05:03.305394 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305364 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" Jul 15 17:05:03.305394 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305372 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" Jul 15 17:05:03.305394 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305380 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" Jul 15 17:05:03.305394 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305386 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" Jul 15 17:05:03.305569 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305466 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13d7124-ea5b-48cf-97c3-f3390e1f0b9c" containerName="test-curl-7410-139944546657312" Jul 15 17:05:03.305569 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305476 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" Jul 15 17:05:03.305569 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.305483 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a0a30a-8c0d-44b2-8e7d-6ce34c0799c0" containerName="istio-proxy" Jul 15 17:05:03.310412 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.310394 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.318378 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.318359 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4vtsf\"/\"kube-root-ca.crt\"" Jul 15 17:05:03.318501 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.318396 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4vtsf\"/\"default-dockercfg-cgjtb\"" Jul 15 17:05:03.319499 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.319483 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4vtsf\"/\"openshift-service-ca.crt\"" Jul 15 17:05:03.336447 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.336413 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4vtsf/must-gather-w5fvz"] Jul 15 17:05:03.388679 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.388635 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dee505-69bb-4d43-85e6-78cfa11c8dbd-must-gather-output\") pod \"must-gather-w5fvz\" (UID: \"65dee505-69bb-4d43-85e6-78cfa11c8dbd\") " pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.388926 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.388759 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66sv\" (UniqueName: \"kubernetes.io/projected/65dee505-69bb-4d43-85e6-78cfa11c8dbd-kube-api-access-w66sv\") pod \"must-gather-w5fvz\" (UID: \"65dee505-69bb-4d43-85e6-78cfa11c8dbd\") " pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.490364 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.490324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dee505-69bb-4d43-85e6-78cfa11c8dbd-must-gather-output\") pod \"must-gather-w5fvz\" (UID: \"65dee505-69bb-4d43-85e6-78cfa11c8dbd\") " pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.490539 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.490406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66sv\" (UniqueName: \"kubernetes.io/projected/65dee505-69bb-4d43-85e6-78cfa11c8dbd-kube-api-access-w66sv\") pod \"must-gather-w5fvz\" (UID: \"65dee505-69bb-4d43-85e6-78cfa11c8dbd\") " pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.490679 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.490658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dee505-69bb-4d43-85e6-78cfa11c8dbd-must-gather-output\") pod \"must-gather-w5fvz\" (UID: \"65dee505-69bb-4d43-85e6-78cfa11c8dbd\") " pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.520121 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.520091 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66sv\" (UniqueName: \"kubernetes.io/projected/65dee505-69bb-4d43-85e6-78cfa11c8dbd-kube-api-access-w66sv\") pod \"must-gather-w5fvz\" (UID: \"65dee505-69bb-4d43-85e6-78cfa11c8dbd\") " pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.619714 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.619683 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4vtsf/must-gather-w5fvz" Jul 15 17:05:03.756334 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.756305 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4vtsf/must-gather-w5fvz"] Jul 15 17:05:03.757462 ip-10-0-136-51 kubenswrapper[2560]: W0715 17:05:03.757437 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65dee505_69bb_4d43_85e6_78cfa11c8dbd.slice/crio-0759b566c06caa2a8f79c99338dea85a9ad296cfadb9453fdb25ba0330f22dcf WatchSource:0}: Error finding container 0759b566c06caa2a8f79c99338dea85a9ad296cfadb9453fdb25ba0330f22dcf: Status 404 returned error can't find the container with id 0759b566c06caa2a8f79c99338dea85a9ad296cfadb9453fdb25ba0330f22dcf Jul 15 17:05:03.836186 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:03.836150 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4vtsf/must-gather-w5fvz" event={"ID":"65dee505-69bb-4d43-85e6-78cfa11c8dbd","Type":"ContainerStarted","Data":"0759b566c06caa2a8f79c99338dea85a9ad296cfadb9453fdb25ba0330f22dcf"} Jul 15 17:05:04.847788 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:04.847548 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4vtsf/must-gather-w5fvz" event={"ID":"65dee505-69bb-4d43-85e6-78cfa11c8dbd","Type":"ContainerStarted","Data":"72d516c74133a49d0f4deec6d68f29f918f9ff8b82bf835c8b2c5ac87aab78bd"} Jul 15 17:05:04.847788 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:04.847586 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4vtsf/must-gather-w5fvz" event={"ID":"65dee505-69bb-4d43-85e6-78cfa11c8dbd","Type":"ContainerStarted","Data":"c379256df075db866d3eb6c27b9bcd876a6e93432d0655da980357d0d504f181"} Jul 15 17:05:04.867585 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:04.867472 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4vtsf/must-gather-w5fvz" podStartSLOduration=1.008376314 podStartE2EDuration="1.867457862s" totalImagesPullingTime="859.081548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 17:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 17:05:04.864356543 +0000 UTC m=+2907.119256763" watchObservedRunningTime="2026-07-15 17:05:04.867457862 +0000 UTC m=+2907.122358067" Jul 15 17:05:06.191907 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:06.191851 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ks4kq_7b77867d-4145-4515-945f-7a50589dd046/global-pull-secret-syncer/0.log" Jul 15 17:05:06.328763 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:06.328733 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rcbg5_7ccd6b87-90bf-4d5e-86d4-867ac7bd23f9/konnectivity-agent/0.log" Jul 15 17:05:06.404394 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:06.404362 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-51.ec2.internal_0220f44d9b57d66146d6849b5a148314/haproxy/0.log" Jul 15 17:05:11.123978 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.123936 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r_c7ac79c2-4a48-4b7d-891d-6f706f2a6466/extract/0.log" Jul 15 17:05:11.166980 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.166946 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r_c7ac79c2-4a48-4b7d-891d-6f706f2a6466/util/0.log" Jul 15 17:05:11.191590 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.191537 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759krj8r_c7ac79c2-4a48-4b7d-891d-6f706f2a6466/pull/0.log" Jul 15 17:05:11.223728 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.223696 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw_1ab4cfc6-39e2-46e5-8161-feca59d8b6e0/extract/0.log" Jul 15 17:05:11.247791 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.247762 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw_1ab4cfc6-39e2-46e5-8161-feca59d8b6e0/util/0.log" Jul 15 17:05:11.276253 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.276221 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0w7rsw_1ab4cfc6-39e2-46e5-8161-feca59d8b6e0/pull/0.log" Jul 15 17:05:11.308990 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.308956 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx_0797ecaa-02dc-4f8a-a374-49ec62364a19/extract/0.log" Jul 15 17:05:11.335790 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.335764 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx_0797ecaa-02dc-4f8a-a374-49ec62364a19/util/0.log" Jul 15 17:05:11.366127 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.366093 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vwgmx_0797ecaa-02dc-4f8a-a374-49ec62364a19/pull/0.log" Jul 15 17:05:11.412078 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.411991 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884_3e350d22-7902-4628-9cae-469398433e27/extract/0.log" Jul 15 17:05:11.456222 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.456174 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884_3e350d22-7902-4628-9cae-469398433e27/util/0.log" Jul 15 17:05:11.475232 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.475206 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1px884_3e350d22-7902-4628-9cae-469398433e27/pull/0.log" Jul 15 17:05:11.914806 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:11.913163 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65c78cc48c-2hz6l_6e3f41a8-c4e5-4a1f-b4c7-d0f9cec2a3fe/authorino/0.log" Jul 15 17:05:12.002530 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:12.002503 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rvdb7_9b4b9e23-66e1-47f9-8145-db66dbebb217/kuadrant-console-plugin/0.log" Jul 15 17:05:13.960056 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:13.960019 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/alertmanager/0.log" Jul 15 17:05:13.987388 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:13.987324 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/config-reloader/0.log" Jul 15 17:05:14.010474 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.010388 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/kube-rbac-proxy-web/0.log" Jul 15 17:05:14.035420 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.035365 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/kube-rbac-proxy/0.log" Jul 15 17:05:14.060171 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.060098 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/kube-rbac-proxy-metric/0.log" Jul 15 17:05:14.085603 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.085569 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/prom-label-proxy/0.log" Jul 15 17:05:14.110085 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.110048 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_85b15886-8010-49c4-9fce-7ceca9102af9/init-config-reloader/0.log" Jul 15 17:05:14.273721 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.273627 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74ff49cf57-t8fcj_4a7778ad-0e65-487d-af6d-35271493696c/metrics-server/0.log" Jul 15 17:05:14.304355 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.304328 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-nwd74_80c12a20-7f52-4f3b-93ba-e71167040c45/monitoring-plugin/0.log" Jul 15 17:05:14.487536 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.487501 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjblt_99bfa9e6-40d0-4685-925f-49493b61683b/node-exporter/0.log" Jul 15 17:05:14.511367 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.511334 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjblt_99bfa9e6-40d0-4685-925f-49493b61683b/kube-rbac-proxy/0.log" Jul 15 17:05:14.536399 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.536316 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjblt_99bfa9e6-40d0-4685-925f-49493b61683b/init-textfile/0.log" Jul 15 17:05:14.567635 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.567595 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-jqj6d_3041f42a-b504-4229-ae91-038d98c9bae2/kube-rbac-proxy-main/0.log" Jul 15 17:05:14.590177 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.590147 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-jqj6d_3041f42a-b504-4229-ae91-038d98c9bae2/kube-rbac-proxy-self/0.log" Jul 15 17:05:14.626148 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.626116 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-jqj6d_3041f42a-b504-4229-ae91-038d98c9bae2/openshift-state-metrics/0.log" Jul 15 17:05:14.826912 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.826813 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ttl9v_42c103cd-610d-49d1-8a0a-a652c7d9261d/prometheus-operator/0.log" Jul 15 17:05:14.847691 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.847664 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ttl9v_42c103cd-610d-49d1-8a0a-a652c7d9261d/kube-rbac-proxy/0.log" Jul 15 17:05:14.904459 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.904423 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc877f5db-wdhq7_e39255c0-38b0-40ed-896f-bf7b212abddd/telemeter-client/0.log" Jul 15 17:05:14.929762 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.929730 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc877f5db-wdhq7_e39255c0-38b0-40ed-896f-bf7b212abddd/reload/0.log" Jul 15 17:05:14.952790 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.952762 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc877f5db-wdhq7_e39255c0-38b0-40ed-896f-bf7b212abddd/kube-rbac-proxy/0.log" Jul 15 17:05:14.983682 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:14.983656 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8f5586cc-gfpkk_4eca1df6-3d6d-480b-bbee-abfb87e4bece/thanos-query/0.log" Jul 15 17:05:15.012911 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:15.012878 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8f5586cc-gfpkk_4eca1df6-3d6d-480b-bbee-abfb87e4bece/kube-rbac-proxy-web/0.log" Jul 15 17:05:15.038100 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:15.038073 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8f5586cc-gfpkk_4eca1df6-3d6d-480b-bbee-abfb87e4bece/kube-rbac-proxy/0.log" Jul 15 17:05:15.068411 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:15.068384 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8f5586cc-gfpkk_4eca1df6-3d6d-480b-bbee-abfb87e4bece/prom-label-proxy/0.log" Jul 15 17:05:15.093614 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:15.093584 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8f5586cc-gfpkk_4eca1df6-3d6d-480b-bbee-abfb87e4bece/kube-rbac-proxy-rules/0.log" Jul 15 17:05:15.124947 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:15.124915 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8f5586cc-gfpkk_4eca1df6-3d6d-480b-bbee-abfb87e4bece/kube-rbac-proxy-metrics/0.log" Jul 15 17:05:16.031527 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.031494 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb"] Jul 15 17:05:16.037275 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.037247 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.044245 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.044214 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb"] Jul 15 17:05:16.127649 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.127612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-lib-modules\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.128020 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.128000 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-proc\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.128180 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.128164 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-sys\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.128359 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.128347 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-podres\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.128501 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.128486 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glh5f\" (UniqueName: \"kubernetes.io/projected/bd3735ec-50e3-46ec-8561-6ba34f64965f-kube-api-access-glh5f\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.229601 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229559 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-podres\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.229798 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229621 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glh5f\" (UniqueName: \"kubernetes.io/projected/bd3735ec-50e3-46ec-8561-6ba34f64965f-kube-api-access-glh5f\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.229798 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229686 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-lib-modules\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.229798 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229755 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-podres\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.229798 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-proc\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.230077 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229816 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-proc\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.230077 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229825 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-sys\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.230077 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.229961 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-sys\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.230270 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.230097 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd3735ec-50e3-46ec-8561-6ba34f64965f-lib-modules\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.239471 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.239441 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glh5f\" (UniqueName: \"kubernetes.io/projected/bd3735ec-50e3-46ec-8561-6ba34f64965f-kube-api-access-glh5f\") pod \"perf-node-gather-daemonset-rz9kb\" (UID: \"bd3735ec-50e3-46ec-8561-6ba34f64965f\") " pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.353605 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.353572 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.545215 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.544631 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb"] Jul 15 17:05:16.548619 ip-10-0-136-51 kubenswrapper[2560]: W0715 17:05:16.548563 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd3735ec_50e3_46ec_8561_6ba34f64965f.slice/crio-7e47793b8bdea9975cdeccc5fb23a25ce88b73a552fdfc0cf92dabd69efd3550 WatchSource:0}: Error finding container 7e47793b8bdea9975cdeccc5fb23a25ce88b73a552fdfc0cf92dabd69efd3550: Status 404 returned error can't find the container with id 7e47793b8bdea9975cdeccc5fb23a25ce88b73a552fdfc0cf92dabd69efd3550 Jul 15 17:05:16.958643 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.958554 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" event={"ID":"bd3735ec-50e3-46ec-8561-6ba34f64965f","Type":"ContainerStarted","Data":"c05b2e4779b8daac6b49b7b1c3c5b24bd250c1faac7bcd6c0d561891a51a4c14"} Jul 15 17:05:16.958643 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.958624 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" event={"ID":"bd3735ec-50e3-46ec-8561-6ba34f64965f","Type":"ContainerStarted","Data":"7e47793b8bdea9975cdeccc5fb23a25ce88b73a552fdfc0cf92dabd69efd3550"} Jul 15 17:05:16.958856 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.958656 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:16.979810 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:16.979760 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" podStartSLOduration=0.979744799 podStartE2EDuration="979.744799ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 17:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 17:05:16.977414277 +0000 UTC m=+2919.232314510" watchObservedRunningTime="2026-07-15 17:05:16.979744799 +0000 UTC m=+2919.234645017" Jul 15 17:05:17.439185 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:17.439155 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55c8c478b8-zs4z5_a78c234c-6a67-425b-bcad-20a77605d0e1/console/0.log" Jul 15 17:05:19.086437 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:19.086409 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rttt_12f6fde1-e731-482e-b77c-8d19635faa93/dns/0.log" Jul 15 17:05:19.110462 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:19.110424 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rttt_12f6fde1-e731-482e-b77c-8d19635faa93/kube-rbac-proxy/0.log" Jul 15 17:05:19.266886 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:19.266842 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-78lgd_f9437532-8ae5-4ded-b3c5-987dffce18fa/dns-node-resolver/0.log" Jul 15 17:05:19.905343 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:19.905312 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-495xf_af3759db-0951-4fdd-8f5d-b5ec058cbb3a/node-ca/0.log" Jul 15 17:05:20.901838 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:20.901805 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pp2zw_b172f886-2c3c-4d70-b50d-51f163e507d8/istio-proxy/0.log" Jul 15 17:05:21.141057 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:21.141010 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84484457c8-qx84h_b0bf833c-4532-4ce5-a10a-4d100a580207/kube-auth-proxy/0.log" Jul 15 17:05:22.037101 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:22.037055 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hc5rh_75e64c35-a318-46b4-857b-46b61dbf3e6f/serve-healthcheck-canary/0.log" Jul 15 17:05:22.674483 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:22.674446 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4fdz_96c0b345-1601-439b-81ba-940e5e8712b4/kube-rbac-proxy/0.log" Jul 15 17:05:22.694430 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:22.694402 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4fdz_96c0b345-1601-439b-81ba-940e5e8712b4/exporter/0.log" Jul 15 17:05:22.716290 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:22.716261 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4fdz_96c0b345-1601-439b-81ba-940e5e8712b4/extractor/0.log" Jul 15 17:05:22.981474 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:22.981395 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4vtsf/perf-node-gather-daemonset-rz9kb" Jul 15 17:05:25.232236 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:25.232204 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lvlkg_a60d3362-5052-43eb-8f12-fc6d12f4e1db/server/0.log" Jul 15 17:05:25.319177 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:25.319144 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r7hzm_3d476544-32be-4eed-8dc5-5db817bc7dd4/manager/0.log" Jul 15 17:05:25.345773 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:25.345738 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599cdff4cb-2bp48_81199833-4935-4991-8d8f-7d86dce289ad/manager/0.log" Jul 15 17:05:25.370691 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:25.370663 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599cdff4cb-2bp48_81199833-4935-4991-8d8f-7d86dce289ad/copy-manifests/0.log" Jul 15 17:05:26.198362 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:26.198333 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pp2zw_b172f886-2c3c-4d70-b50d-51f163e507d8/istio-proxy/0.log" Jul 15 17:05:26.422497 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:26.422469 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84484457c8-qx84h_b0bf833c-4532-4ce5-a10a-4d100a580207/kube-auth-proxy/0.log" Jul 15 17:05:26.820272 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:26.820214 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f875d5cd5-wpnws_31af93e2-d84f-4609-b50f-560d6f5ccc0d/manager/0.log" Jul 15 17:05:27.741960 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:27.741882 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pp2zw_b172f886-2c3c-4d70-b50d-51f163e507d8/istio-proxy/0.log" Jul 15 17:05:27.975200 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:27.975156 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84484457c8-qx84h_b0bf833c-4532-4ce5-a10a-4d100a580207/kube-auth-proxy/0.log" Jul 15 17:05:33.629370 ip-10-0-136-51 kubenswrapper[2560]: I0715 17:05:33.629340 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vtfzn_ad2f5186-47e7-41a8-a6c3-91b8093440b3/kube-multus-additional-cni-plugins/0.log"