Jul 23 19:13:53.694375 ip-10-0-133-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 19:13:53.694386 ip-10-0-133-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 19:13:53.694393 ip-10-0-133-134 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 19:13:53.694634 ip-10-0-133-134 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 19:14:03.772291 ip-10-0-133-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 19:14:03.772307 ip-10-0-133-134 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ed6bb180f4f54435b1768b895926ad5e -- Jul 23 19:15:52.328023 ip-10-0-133-134 systemd[1]: Starting Kubernetes Kubelet... Jul 23 19:15:52.654933 ip-10-0-133-134 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 19:15:52.654933 ip-10-0-133-134 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 19:15:52.654933 ip-10-0-133-134 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 19:15:52.654933 ip-10-0-133-134 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 19:15:52.654933 ip-10-0-133-134 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 19:15:52.656669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.656578 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 19:15:52.661325 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661307 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:15:52.661325 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661325 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661329 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661332 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661335 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661337 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661340 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661344 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661347 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661350 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661353 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661363 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661366 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661369 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661371 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661374 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661376 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661379 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661382 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661384 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661387 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:15:52.661409 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661389 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661393 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661398 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661401 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661404 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661406 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661409 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661412 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661415 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661418 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661420 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661423 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661426 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661428 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661431 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661433 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661436 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661439 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661441 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:15:52.661895 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661444 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661448 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661452 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661455 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661460 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661464 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661467 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661470 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661473 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661475 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661478 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661481 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661483 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661486 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661488 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661491 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661494 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661497 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661499 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:15:52.662342 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661502 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661505 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661507 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661510 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661513 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661515 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661518 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661521 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661523 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661526 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661528 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661531 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661534 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661537 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661539 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661543 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661546 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661548 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661551 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661554 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:15:52.662817 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661556 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661559 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661561 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661563 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661566 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661568 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661571 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661573 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661576 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661580 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661582 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661585 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661587 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661590 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661592 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661595 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661597 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661600 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661602 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.661605 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:15:52.663323 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662360 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662367 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662370 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662373 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662375 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662378 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662381 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662384 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662387 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662389 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662394 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662398 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662401 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662404 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662407 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662410 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662413 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662416 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662419 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:15:52.663812 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662422 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662427 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662430 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662432 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662435 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662438 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662440 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662443 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662446 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662449 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662451 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662454 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662457 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662459 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662463 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662466 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662469 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662472 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662474 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662484 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:15:52.664279 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662486 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662489 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662492 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662495 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662497 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662500 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662503 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662505 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662508 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662510 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662513 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662516 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662519 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662522 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662525 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662528 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662530 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662533 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662536 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662538 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:15:52.664771 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662541 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662543 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662546 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662549 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662551 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662554 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662556 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662559 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662561 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662564 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662566 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662569 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662573 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662575 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662578 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662581 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662583 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662586 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662588 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662590 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:15:52.665250 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662593 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662596 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662598 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662601 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662603 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662606 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662608 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662612 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662614 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662618 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662621 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662624 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662627 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662630 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662633 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662635 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662638 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662640 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662643 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.662645 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:15:52.665740 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663065 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663070 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663074 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663077 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663086 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663089 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663091 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663094 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663097 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663100 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663104 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663107 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663110 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663113 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663116 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663119 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663122 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663125 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663127 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663130 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:15:52.666222 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663132 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663135 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663137 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663140 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663142 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663145 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663148 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663150 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663153 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663156 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663158 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663161 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663163 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663166 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663169 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663171 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663173 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663177 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663179 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663182 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:15:52.666702 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663184 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663187 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663189 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663194 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663197 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663200 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663202 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663206 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663209 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663213 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663216 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663219 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663222 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663225 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663228 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663230 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663233 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663236 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663238 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:15:52.667208 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663241 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663243 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663246 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663248 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663253 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663255 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663258 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663261 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663264 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663267 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663269 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663275 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663277 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663280 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663282 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663285 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663287 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663290 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663293 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:15:52.667663 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663295 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663299 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663302 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663304 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663307 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663309 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663311 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663314 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663317 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663319 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663322 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663324 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663327 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663329 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663332 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663335 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663337 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663341 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663344 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663347 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:15:52.668191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.663349 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663428 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663438 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663454 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663460 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663470 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663476 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663482 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663487 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663490 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663493 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663497 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663501 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663505 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663508 2574 flags.go:64] FLAG: --cgroup-root="" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663511 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663514 2574 flags.go:64] FLAG: --client-ca-file="" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663517 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663520 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663525 2574 flags.go:64] FLAG: --cluster-domain="" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663528 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663531 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663535 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663538 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 19:15:52.668675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663542 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663545 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663548 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663552 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663555 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663559 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663562 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663565 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663568 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663572 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663575 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663578 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663581 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663584 2574 flags.go:64] FLAG: --enable-server="true" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663587 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663591 2574 flags.go:64] FLAG: --event-burst="100" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663595 2574 flags.go:64] FLAG: --event-qps="50" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663598 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663601 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663604 2574 flags.go:64] FLAG: --eviction-hard="" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663607 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663611 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663614 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663617 2574 flags.go:64] FLAG: --eviction-soft="" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663620 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 19:15:52.669258 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663623 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663625 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663628 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663631 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663634 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663637 2574 flags.go:64] FLAG: --feature-gates="" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663644 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663647 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663651 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663654 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663657 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663660 2574 flags.go:64] FLAG: --help="false" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663664 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663667 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663671 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663673 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663677 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663680 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663683 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663686 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663688 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663691 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663695 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663698 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 19:15:52.669878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663701 2574 flags.go:64] FLAG: --kube-reserved="" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663704 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663706 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663709 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663714 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663717 2574 flags.go:64] FLAG: --lock-file="" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663719 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663722 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663725 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663743 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663746 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663749 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663751 2574 flags.go:64] FLAG: --logging-format="text" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663754 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663757 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663760 2574 flags.go:64] FLAG: --manifest-url="" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663764 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663768 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663771 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663775 2574 flags.go:64] FLAG: --max-pods="110" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663779 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663782 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663785 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663788 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663791 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 19:15:52.670798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663794 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663797 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663805 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663808 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663811 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663814 2574 flags.go:64] FLAG: --pod-cidr="" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663818 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663824 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663826 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663840 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663843 2574 flags.go:64] FLAG: --port="10250" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663848 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663851 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b093c5b8906a0eff" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663854 2574 flags.go:64] FLAG: --qos-reserved="" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663857 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663860 2574 flags.go:64] FLAG: --register-node="true" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663863 2574 flags.go:64] FLAG: --register-with-taints="" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663867 2574 flags.go:64] FLAG: --registry-burst="10" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663870 2574 flags.go:64] FLAG: --registry-qps="5" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663872 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663875 2574 flags.go:64] FLAG: --reserved-memory="" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663879 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663882 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663884 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663887 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 19:15:52.671877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663890 2574 flags.go:64] FLAG: --runonce="false" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663893 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663896 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663900 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663903 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663906 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663909 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663913 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663916 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663919 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663921 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663924 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663927 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663930 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663934 2574 flags.go:64] FLAG: --system-cgroups="" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663936 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663942 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663945 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663949 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663953 2574 flags.go:64] FLAG: --tls-min-version="" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663956 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663959 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663962 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663965 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663967 2574 flags.go:64] FLAG: --v="2" Jul 23 19:15:52.672780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663971 2574 flags.go:64] FLAG: --version="false" Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663975 2574 flags.go:64] FLAG: --vmodule="" Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663979 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.663982 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664110 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664114 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664116 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664119 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664122 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664124 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664129 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664131 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664134 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664136 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664141 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664144 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664147 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664150 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664153 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664156 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:15:52.673659 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664159 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664162 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664166 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664168 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664171 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664174 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664177 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664179 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664182 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664185 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664187 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664189 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664192 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664195 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664197 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664200 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664203 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664205 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664208 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664210 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:15:52.674523 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664213 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664215 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664219 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664221 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664224 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664226 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664229 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664231 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664235 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664237 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664240 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664243 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664245 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664248 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664251 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664254 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664257 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664260 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664263 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:15:52.675098 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664265 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664268 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664270 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664273 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664275 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664278 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664281 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664284 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664287 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664290 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664292 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664295 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664297 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664300 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664303 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664306 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664309 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664312 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664315 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664317 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664320 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:15:52.675934 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664322 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664325 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664327 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664330 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664332 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664335 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664337 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664340 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664343 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664348 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664350 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664353 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664355 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664359 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664363 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664366 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664369 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664371 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664374 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:15:52.676688 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664377 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664381 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664384 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.664387 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.664994 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.672363 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.672383 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.673068 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.673083 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673190 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673198 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673202 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673208 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673212 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673218 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:15:52.677203 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673222 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673227 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673232 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673236 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673240 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673244 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673249 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673254 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673258 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673263 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673267 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673271 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673276 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673280 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673284 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673288 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673293 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673297 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673301 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673305 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:15:52.677651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673310 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673314 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673318 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673324 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673329 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673334 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673338 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673342 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673347 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673351 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673355 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673359 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673364 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673367 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673372 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673376 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673381 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673387 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673394 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:15:52.678299 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673398 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673403 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673407 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673411 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673415 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673419 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673424 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673428 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673432 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673437 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673441 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673445 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673449 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673453 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673458 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673462 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673467 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673472 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673477 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673481 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:15:52.678944 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673485 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673489 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673493 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673498 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673502 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673506 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673510 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673514 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673518 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673522 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673526 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673531 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673534 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673539 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673543 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673548 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673552 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673556 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673561 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673565 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:15:52.679493 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673569 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673573 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673577 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673581 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673585 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673590 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673594 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673598 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673603 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673608 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673613 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673617 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673623 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673630 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:15:52.680166 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.673638 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673838 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673849 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673856 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673863 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673868 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673873 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673878 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673884 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673890 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673894 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673899 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673905 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673909 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673914 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673919 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673923 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673928 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673933 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673937 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 19:15:52.680514 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673942 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673947 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673951 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673956 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673961 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673965 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673969 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673974 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673979 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673984 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673988 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673993 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.673997 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674001 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674006 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674010 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674014 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674018 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674023 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 19:15:52.681050 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674027 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674031 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674035 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674039 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674043 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674047 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674052 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674058 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674064 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674069 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674074 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674078 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674083 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674088 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674092 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674095 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674100 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674104 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674109 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674113 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 19:15:52.681778 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674118 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674122 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674127 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674131 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674136 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674140 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674145 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674149 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674153 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674157 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674161 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674165 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674170 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674174 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674178 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674181 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674186 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674190 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674194 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 19:15:52.682380 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674198 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674202 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674206 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674210 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674214 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674218 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674222 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674226 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674231 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674236 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674240 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674244 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674248 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674252 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674257 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674261 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674264 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674270 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674275 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674278 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 19:15:52.682902 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674282 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:52.674286 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.674295 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.674454 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.677991 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.679291 2574 server.go:1019] "Starting client certificate rotation" Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.679393 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.679436 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 19:15:52.683744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.681797 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 19:15:52.686464 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.686442 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 19:15:52.698327 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.698308 2574 log.go:25] "Validated CRI v1 runtime API" Jul 23 19:15:52.700301 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.700288 2574 log.go:25] "Validated CRI v1 image API" Jul 23 19:15:52.703148 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.703127 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 19:15:52.705300 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.705279 2574 fs.go:135] Filesystem UUIDs: map[1499a686-64c3-4229-a873-ac53a8d4925a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9562bc98-f1e2-49bc-90c0-922f271ce0de:/dev/nvme0n1p4] Jul 23 19:15:52.705353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.705299 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 19:15:52.710605 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.710491 2574 manager.go:216] Machine: {Timestamp:2026-07-23 19:15:52.709295781 +0000 UTC m=+0.281916312 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097185 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec203a4c5470f88c1413eb202efe34f4 SystemUUID:ec203a4c-5470-f88c-1413-eb202efe34f4 BootID:ed6bb180-f4f5-4435-b176-8b895926ad5e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a9:f6:2d:66:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a9:f6:2d:66:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:05:af:a7:29:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 19:15:52.710605 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.710599 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 19:15:52.710725 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.710685 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 19:15:52.713426 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.713394 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 19:15:52.713571 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.713428 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 19:15:52.714121 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.714111 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 19:15:52.714162 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.714124 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 19:15:52.714162 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.714131 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 19:15:52.715246 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.715233 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 19:15:52.716599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.716589 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 19:15:52.718379 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.718351 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 19:15:52.720624 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720610 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 23 19:15:52.720840 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720829 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 19:15:52.720885 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720862 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 23 19:15:52.720885 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720875 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 19:15:52.720955 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720891 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 19:15:52.720955 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720905 2574 kubelet.go:398] "Adding apiserver pod source" Jul 23 19:15:52.720955 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.720914 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 19:15:52.721444 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.721430 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 19:15:52.721509 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.721458 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 19:15:52.723315 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.723300 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 19:15:52.725213 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.725200 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 19:15:52.725255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.725230 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 19:15:52.726415 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726402 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 19:15:52.726478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726421 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 19:15:52.726478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726427 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 19:15:52.726478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726433 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 19:15:52.726478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726442 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 19:15:52.726478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726464 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 19:15:52.726478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726477 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 19:15:52.726646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726486 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 19:15:52.726646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726496 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 19:15:52.726646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726505 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 19:15:52.726646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726527 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 19:15:52.726646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.726538 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 19:15:52.727217 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.727204 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 19:15:52.727248 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.727218 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 19:15:52.730574 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.730554 2574 server.go:1268] "Started kubelet" Jul 23 19:15:52.733910 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.733871 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 19:15:52.734060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.734044 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 19:15:52.734686 ip-10-0-133-134 systemd[1]: Started Kubernetes Kubelet. Jul 23 19:15:52.735287 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.734873 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 19:15:52.735287 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.735097 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 19:15:52.735500 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.735312 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 19:15:52.735500 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.735443 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 19:15:52.736777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.736723 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 19:15:52.738233 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.738216 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 23 19:15:52.739344 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.739320 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 19:15:52.743498 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.742634 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-134.ec2.internal.18c50134987ccad9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-134.ec2.internal,UID:ip-10-0-133-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-134.ec2.internal,},FirstTimestamp:2026-07-23 19:15:52.730512089 +0000 UTC m=+0.303132620,LastTimestamp:2026-07-23 19:15:52.730512089 +0000 UTC m=+0.303132620,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-134.ec2.internal,}" Jul 23 19:15:52.744129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744108 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 19:15:52.744129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744117 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 19:15:52.744281 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.744216 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:52.744726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744704 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 19:15:52.744726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744706 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 19:15:52.744873 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744751 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 19:15:52.744873 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744824 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 23 19:15:52.744873 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.744835 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 23 19:15:52.745533 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745512 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 19:15:52.745533 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745534 2574 factory.go:55] Registering systemd factory Jul 23 19:15:52.745711 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745544 2574 factory.go:223] Registration of the systemd container factory successfully Jul 23 19:15:52.745812 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745798 2574 factory.go:153] Registering CRI-O factory Jul 23 19:15:52.745865 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745813 2574 factory.go:223] Registration of the crio container factory successfully Jul 23 19:15:52.745865 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745846 2574 factory.go:103] Registering Raw factory Jul 23 19:15:52.745865 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.745860 2574 manager.go:1196] Started watching for new ooms in manager Jul 23 19:15:52.746278 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.746263 2574 manager.go:349] Starting recovery of all containers Jul 23 19:15:52.747939 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.747914 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 19:15:52.748096 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.748073 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 19:15:52.753215 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.753142 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 19:15:52.756845 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.756825 2574 manager.go:354] Recovery completed Jul 23 19:15:52.758413 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.758395 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 23 19:15:52.763391 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.763370 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.765342 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765315 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.765424 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765357 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.765424 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765370 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.765965 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765946 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 19:15:52.765965 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765965 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 19:15:52.766099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765985 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 23 19:15:52.766099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.765956 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.766812 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.766796 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.766867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.766820 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.766867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.766829 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.768144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.768120 2574 policy_none.go:49] "None policy: Start" Jul 23 19:15:52.768144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.768147 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 19:15:52.768266 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.768161 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 19:15:52.768266 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.768136 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-134.ec2.internal.18c501349a903f6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-134.ec2.internal,UID:ip-10-0-133-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-134.ec2.internal,},FirstTimestamp:2026-07-23 19:15:52.765341548 +0000 UTC m=+0.337962079,LastTimestamp:2026-07-23 19:15:52.765341548 +0000 UTC m=+0.337962079,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-134.ec2.internal,}" Jul 23 19:15:52.770153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.770140 2574 policy_none.go:47] "Start" Jul 23 19:15:52.781059 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.780850 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-134.ec2.internal.18c501349a90984e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-134.ec2.internal,UID:ip-10-0-133-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-134.ec2.internal,},FirstTimestamp:2026-07-23 19:15:52.765364302 +0000 UTC m=+0.337984834,LastTimestamp:2026-07-23 19:15:52.765364302 +0000 UTC m=+0.337984834,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-134.ec2.internal,}" Jul 23 19:15:52.783933 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.783863 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-134.ec2.internal.18c501349a90c157 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-134.ec2.internal,UID:ip-10-0-133-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-134.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-134.ec2.internal,},FirstTimestamp:2026-07-23 19:15:52.765374807 +0000 UTC m=+0.337995338,LastTimestamp:2026-07-23 19:15:52.765374807 +0000 UTC m=+0.337995338,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-134.ec2.internal,}" Jul 23 19:15:52.789254 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.789222 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z29cm" Jul 23 19:15:52.795137 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.795031 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-134.ec2.internal.18c501349a903f6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-134.ec2.internal.18c501349a903f6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-134.ec2.internal,UID:ip-10-0-133-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-134.ec2.internal,},FirstTimestamp:2026-07-23 19:15:52.765341548 +0000 UTC m=+0.337962079,LastTimestamp:2026-07-23 19:15:52.766809215 +0000 UTC m=+0.339429745,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-134.ec2.internal,}" Jul 23 19:15:52.802824 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.802788 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z29cm" Jul 23 19:15:52.806927 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.806859 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-134.ec2.internal.18c501349a90984e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-134.ec2.internal.18c501349a90984e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-134.ec2.internal,UID:ip-10-0-133-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-134.ec2.internal,},FirstTimestamp:2026-07-23 19:15:52.765364302 +0000 UTC m=+0.337984834,LastTimestamp:2026-07-23 19:15:52.766824948 +0000 UTC m=+0.339445478,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-134.ec2.internal,}" Jul 23 19:15:52.810513 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.810496 2574 manager.go:337] "Starting Device Plugin manager" Jul 23 19:15:52.810599 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.810530 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 19:15:52.810599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.810541 2574 server.go:87] "Starting device plugin registration server" Jul 23 19:15:52.810844 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.810830 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 19:15:52.810914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.810847 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 19:15:52.810976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.810956 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 19:15:52.811176 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.811066 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 19:15:52.811176 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.811081 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 19:15:52.811761 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.811733 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 19:15:52.811830 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.811798 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:52.840367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.840341 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 19:15:52.849567 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.840388 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 19:15:52.849567 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.840412 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 19:15:52.849567 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.840448 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 19:15:52.849567 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.842959 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:15:52.915805 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.915701 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.916885 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.916864 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.917001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.916906 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.917001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.916922 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.917001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.916956 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.924421 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.924401 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.924508 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.924427 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-134.ec2.internal\": node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:52.941469 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.941436 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal"] Jul 23 19:15:52.941596 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.941531 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.942492 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.942473 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.942600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.942504 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.942600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.942514 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.945240 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.945213 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.945328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.945244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/802635c36462f36fb851b8a8323d592f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal\" (UID: \"802635c36462f36fb851b8a8323d592f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.945328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.945278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/802635c36462f36fb851b8a8323d592f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal\" (UID: \"802635c36462f36fb851b8a8323d592f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.945420 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.945348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.945420 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.945379 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.946031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.946015 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.946099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.946039 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.946099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.946050 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.946099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.946066 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.946192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.946066 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.946192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.946178 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.948678 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.948662 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.948744 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.948690 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:52.949519 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.949504 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:52.949622 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.949533 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:52.949622 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:52.949549 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:52.959710 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.959691 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:52.970626 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.970602 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-134.ec2.internal\" not found" node="ip-10-0-133-134.ec2.internal" Jul 23 19:15:52.975358 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:52.975340 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-134.ec2.internal\" not found" node="ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.045378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.045346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/802635c36462f36fb851b8a8323d592f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal\" (UID: \"802635c36462f36fb851b8a8323d592f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.045378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.045379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/802635c36462f36fb851b8a8323d592f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal\" (UID: \"802635c36462f36fb851b8a8323d592f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.045589 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.045402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ade0de55707084dd778646963520c9b6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-134.ec2.internal\" (UID: \"ade0de55707084dd778646963520c9b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.045589 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.045466 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/802635c36462f36fb851b8a8323d592f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal\" (UID: \"802635c36462f36fb851b8a8323d592f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.045589 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.045466 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/802635c36462f36fb851b8a8323d592f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal\" (UID: \"802635c36462f36fb851b8a8323d592f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.060358 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.060326 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.156580 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.156545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ade0de55707084dd778646963520c9b6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-134.ec2.internal\" (UID: \"ade0de55707084dd778646963520c9b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.156739 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.156616 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ade0de55707084dd778646963520c9b6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-134.ec2.internal\" (UID: \"ade0de55707084dd778646963520c9b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.160880 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.160840 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.261211 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.261133 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.274281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.274248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.278348 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.278327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" Jul 23 19:15:53.362393 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.362349 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.463154 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.463110 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.563650 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.563566 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.646150 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.646121 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:15:53.664245 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.664215 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.679508 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.679479 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 19:15:53.679660 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.679637 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 19:15:53.679700 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.679666 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 19:15:53.764456 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.764421 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.808691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.808651 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 19:10:53 +0000 UTC" deadline="2028-01-12 02:56:15.058326325 +0000 UTC" Jul 23 19:15:53.808691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.808688 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12895h40m21.24964236s" Jul 23 19:15:53.864914 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.864887 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.884107 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.884067 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 19:15:53.885469 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.885443 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientMemory" Jul 23 19:15:53.885615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.885483 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 19:15:53.885615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.885496 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeHasSufficientPID" Jul 23 19:15:53.965976 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:53.965937 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:53.976854 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:53.976798 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802635c36462f36fb851b8a8323d592f.slice/crio-34a587c6a6a102e767ed98a31b51f0da3ed20df05a7b6f1e37fb369c785c2c2a WatchSource:0}: Error finding container 34a587c6a6a102e767ed98a31b51f0da3ed20df05a7b6f1e37fb369c785c2c2a: Status 404 returned error can't find the container with id 34a587c6a6a102e767ed98a31b51f0da3ed20df05a7b6f1e37fb369c785c2c2a Jul 23 19:15:53.982356 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:53.982337 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:15:54.033516 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.033481 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:15:54.066322 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.066293 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:54.166835 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.166763 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-134.ec2.internal\" not found" Jul 23 19:15:54.208114 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:54.208081 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade0de55707084dd778646963520c9b6.slice/crio-04ff3a93284f2348a4abc6d263a18907b66d16a7ddfc27984d19c7f0dc7c8535 WatchSource:0}: Error finding container 04ff3a93284f2348a4abc6d263a18907b66d16a7ddfc27984d19c7f0dc7c8535: Status 404 returned error can't find the container with id 04ff3a93284f2348a4abc6d263a18907b66d16a7ddfc27984d19c7f0dc7c8535 Jul 23 19:15:54.256957 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.256918 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:15:54.347188 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.347148 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" Jul 23 19:15:54.361377 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.361342 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 19:15:54.362355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.362334 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" Jul 23 19:15:54.376404 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.376380 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 19:15:54.722016 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.721983 2574 apiserver.go:52] "Watching apiserver" Jul 23 19:15:54.728618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.728587 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 19:15:54.731405 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.731376 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-4xskk","openshift-dns/node-resolver-qc484","openshift-image-registry/node-ca-4xn7q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal","openshift-multus/multus-vdsw8","openshift-multus/network-metrics-daemon-8679f","kube-system/konnectivity-agent-mfjtk","kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727","openshift-multus/multus-additional-cni-plugins-4w5f5","openshift-network-diagnostics/network-check-target-x5szb","openshift-network-operator/iptables-alerter-2jlnd","openshift-ovn-kubernetes/ovnkube-node-49l7x","kube-system/global-pull-secret-syncer-2gqgt"] Jul 23 19:15:54.734285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.734256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.736818 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.736796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.737079 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.737058 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 19:15:54.737079 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.737074 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.737588 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.737496 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 19:15:54.737691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.737613 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 19:15:54.737691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.737678 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg2sw\"" Jul 23 19:15:54.738100 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.738080 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.739086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.739588 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.739754 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.740095 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.740122 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tqrpd\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.740177 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.740393 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 19:15:54.740522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.740413 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.742091 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.742071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.742190 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.742169 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:15:54.743602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.742918 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hnvp\"" Jul 23 19:15:54.744172 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.744146 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.744367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.744325 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 19:15:54.744605 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.744584 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.745202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.745180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.745202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.745198 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 19:15:54.747661 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.747637 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 19:15:54.748020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.747998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:54.748129 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.748100 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:15:54.748331 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.748303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r4brz\"" Jul 23 19:15:54.751514 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.751494 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.754094 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.754074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.754577 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.754558 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.755104 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.754817 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 19:15:54.755104 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.754824 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp2r2\"" Jul 23 19:15:54.755104 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.754906 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.756524 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.756504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.756741 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.756722 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 19:15:54.757118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.757097 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 19:15:54.757327 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.757309 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dl4vv\"" Jul 23 19:15:54.759523 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.759502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.759620 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.759531 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.759708 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.759688 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pztv\"" Jul 23 19:15:54.759890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.759872 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.762108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.762087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:54.762211 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.762165 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:15:54.762874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.762854 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mw24f\"" Jul 23 19:15:54.762986 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.762967 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.763735 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.763716 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 19:15:54.763735 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.763718 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.765542 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.765522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.766876 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.766854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.766958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.766893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.766958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.766923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-cni-multus\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767066 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.766966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-hostroot\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767066 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-daemon-config\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767066 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-os-release\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.767066 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-var-lib-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-conf-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-multus-certs\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-etc-kubernetes\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sb58\" (UniqueName: \"kubernetes.io/projected/106b3961-d9dd-4575-8a01-01e11abaae6c-kube-api-access-9sb58\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767408 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-cni-netd\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767408 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.767408 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-systemd-units\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767408 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767388 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-etc-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-run-ovn-kubernetes\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d4319b6-feb5-4089-8896-3e93d2e2e099-host\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.767602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccx7\" (UniqueName: \"kubernetes.io/projected/8d4319b6-feb5-4089-8896-3e93d2e2e099-kube-api-access-mccx7\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.767602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-system-cni-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106b3961-d9dd-4575-8a01-01e11abaae6c-cni-binary-copy\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-k8s-cni-cncf-io\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-netns\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-cni-bin\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-kubelet\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-run-netns\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.767864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-system-cni-dir\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6pt\" (UniqueName: \"kubernetes.io/projected/9b32df0e-06b0-4133-b367-edcf67395c90-kube-api-access-5l6pt\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-systemd\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767988 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768003 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768024 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.767988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-log-socket\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-ovnkube-config\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b0171d6-8abc-4a3e-a72a-f29271153a70-kubelet-config\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-socket-dir-parent\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-cni-bin\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-env-overrides\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-cnibin\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-ovn\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-node-log\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768309 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmh8z\"" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9da84ece-26a9-458f-841a-4b08e3060f97-ovn-node-metrics-cert\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt57t\" (UniqueName: \"kubernetes.io/projected/9da84ece-26a9-458f-841a-4b08e3060f97-kube-api-access-qt57t\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-cnibin\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-kubelet\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-slash\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-ovnkube-script-lib\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d4319b6-feb5-4089-8896-3e93d2e2e099-serviceca\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b0171d6-8abc-4a3e-a72a-f29271153a70-dbus\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-cni-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-os-release\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.768880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.768679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.789984 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.789951 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w2ftg" Jul 23 19:15:54.800058 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.800035 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w2ftg" Jul 23 19:15:54.845917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.845837 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 19:15:54.848893 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.848834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" event={"ID":"802635c36462f36fb851b8a8323d592f","Type":"ContainerStarted","Data":"34a587c6a6a102e767ed98a31b51f0da3ed20df05a7b6f1e37fb369c785c2c2a"} Jul 23 19:15:54.850023 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.849996 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" event={"ID":"ade0de55707084dd778646963520c9b6","Type":"ContainerStarted","Data":"04ff3a93284f2348a4abc6d263a18907b66d16a7ddfc27984d19c7f0dc7c8535"} Jul 23 19:15:54.869238 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6pt\" (UniqueName: \"kubernetes.io/projected/9b32df0e-06b0-4133-b367-edcf67395c90-kube-api-access-5l6pt\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.869407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b0171d6-8abc-4a3e-a72a-f29271153a70-kubelet-config\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.869407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-socket-dir-parent\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.869407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-cni-multus\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.869407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869365 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-cni-multus\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.869407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b0171d6-8abc-4a3e-a72a-f29271153a70-kubelet-config\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.869641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869437 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-socket-dir-parent\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.869641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1abb1ad3-cc68-46d5-a07d-d2b4a3918de9-konnectivity-ca\") pod \"konnectivity-agent-mfjtk\" (UID: \"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9\") " pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.869641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-kubernetes\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.869641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-cni-bin\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-env-overrides\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-cnibin\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869684 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-cni-bin\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-ovn\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-cnibin\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9da84ece-26a9-458f-841a-4b08e3060f97-ovn-node-metrics-cert\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-ovn\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt57t\" (UniqueName: \"kubernetes.io/projected/9da84ece-26a9-458f-841a-4b08e3060f97-kube-api-access-qt57t\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-conf-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-kubelet\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-ovnkube-script-lib\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869945 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-conf-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-kubelet\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.869981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d4319b6-feb5-4089-8896-3e93d2e2e099-serviceca\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-cni-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-os-release\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-hostroot\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-daemon-config\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8884e0ca-e537-474d-accd-3a698941907b-host-slash\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-os-release\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870182 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-cni-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-hostroot\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870279 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-env-overrides\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-hosts-file\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-multus-certs\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.870345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-sys\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-system-cni-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106b3961-d9dd-4575-8a01-01e11abaae6c-cni-binary-copy\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-k8s-cni-cncf-io\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-netns\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-multus-certs\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-etc-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d4319b6-feb5-4089-8896-3e93d2e2e099-serviceca\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870497 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-ovnkube-script-lib\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d4319b6-feb5-4089-8896-3e93d2e2e099-host\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-k8s-cni-cncf-io\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccx7\" (UniqueName: \"kubernetes.io/projected/8d4319b6-feb5-4089-8896-3e93d2e2e099-kube-api-access-mccx7\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d4319b6-feb5-4089-8896-3e93d2e2e099-host\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-etc-selinux\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-etc-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-system-cni-dir\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-kubelet\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysconfig\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.871226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870689 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-kubelet\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-run-netns\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-systemd\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870726 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-tuned\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmzr\" (UniqueName: \"kubernetes.io/projected/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-kube-api-access-zmmzr\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-run-netns\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-device-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-host\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-run-netns\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106b3961-d9dd-4575-8a01-01e11abaae6c-multus-daemon-config\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-system-cni-dir\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.870967 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-systemd\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-system-cni-dir\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.872118 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-log-socket\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871083 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-run-systemd\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871109 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106b3961-d9dd-4575-8a01-01e11abaae6c-cni-binary-copy\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-ovnkube-config\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-log-socket\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-node-log\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-cnibin\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sb58\" (UniqueName: \"kubernetes.io/projected/106b3961-d9dd-4575-8a01-01e11abaae6c-kube-api-access-9sb58\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-node-log\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-slash\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871438 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-cnibin\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b0171d6-8abc-4a3e-a72a-f29271153a70-dbus\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871504 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-slash\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-sys-fs\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b0171d6-8abc-4a3e-a72a-f29271153a70-dbus\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9da84ece-26a9-458f-841a-4b08e3060f97-ovnkube-config\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.872890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.871585 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-registration-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kd2\" (UniqueName: \"kubernetes.io/projected/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-kube-api-access-86kd2\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq86\" (UniqueName: \"kubernetes.io/projected/2d89f060-5022-4975-8fa5-2b17fb96675a-kube-api-access-6jq86\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.871789 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:15:55.371728227 +0000 UTC m=+2.944348761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5g7\" (UniqueName: \"kubernetes.io/projected/8884e0ca-e537-474d-accd-3a698941907b-kube-api-access-nk5g7\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871871 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-os-release\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-var-lib-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-etc-kubernetes\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-var-lib-kubelet\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.871981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-os-release\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-var-lib-openvswitch\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-socket-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-cni-binary-copy\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.873600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-cni-netd\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872072 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-etc-kubernetes\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-systemd-units\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-cni-netd\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-systemd-units\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-run-ovn-kubernetes\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-run-ovn-kubernetes\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-tmp-dir\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1abb1ad3-cc68-46d5-a07d-d2b4a3918de9-agent-certs\") pod \"konnectivity-agent-mfjtk\" (UID: \"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9\") " pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b32df0e-06b0-4133-b367-edcf67395c90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-cni-bin\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b32df0e-06b0-4133-b367-edcf67395c90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf6z4\" (UniqueName: \"kubernetes.io/projected/f96b8b8d-ba63-475d-9005-660e902fddff-kube-api-access-gf6z4\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-modprobe-d\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysctl-d\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106b3961-d9dd-4575-8a01-01e11abaae6c-host-var-lib-cni-bin\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.874332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysctl-conf\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-run\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d89f060-5022-4975-8fa5-2b17fb96675a-tmp\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-lib-modules\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8884e0ca-e537-474d-accd-3a698941907b-iptables-alerter-script\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.872682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da84ece-26a9-458f-841a-4b08e3060f97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.875234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.874387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9da84ece-26a9-458f-841a-4b08e3060f97-ovn-node-metrics-cert\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.883510 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.883482 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6pt\" (UniqueName: \"kubernetes.io/projected/9b32df0e-06b0-4133-b367-edcf67395c90-kube-api-access-5l6pt\") pod \"multus-additional-cni-plugins-4w5f5\" (UID: \"9b32df0e-06b0-4133-b367-edcf67395c90\") " pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:54.884482 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.884441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sb58\" (UniqueName: \"kubernetes.io/projected/106b3961-d9dd-4575-8a01-01e11abaae6c-kube-api-access-9sb58\") pod \"multus-vdsw8\" (UID: \"106b3961-d9dd-4575-8a01-01e11abaae6c\") " pod="openshift-multus/multus-vdsw8" Jul 23 19:15:54.884606 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.884509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt57t\" (UniqueName: \"kubernetes.io/projected/9da84ece-26a9-458f-841a-4b08e3060f97-kube-api-access-qt57t\") pod \"ovnkube-node-49l7x\" (UID: \"9da84ece-26a9-458f-841a-4b08e3060f97\") " pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:54.884606 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.884543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccx7\" (UniqueName: \"kubernetes.io/projected/8d4319b6-feb5-4089-8896-3e93d2e2e099-kube-api-access-mccx7\") pod \"node-ca-4xn7q\" (UID: \"8d4319b6-feb5-4089-8896-3e93d2e2e099\") " pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:54.912538 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.912507 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:15:54.973646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-tmp-dir\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.973646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1abb1ad3-cc68-46d5-a07d-d2b4a3918de9-agent-certs\") pod \"konnectivity-agent-mfjtk\" (UID: \"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9\") " pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.973646 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf6z4\" (UniqueName: \"kubernetes.io/projected/f96b8b8d-ba63-475d-9005-660e902fddff-kube-api-access-gf6z4\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-modprobe-d\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysctl-d\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysctl-conf\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-run\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d89f060-5022-4975-8fa5-2b17fb96675a-tmp\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:54.973953 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-lib-modules\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-modprobe-d\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.973978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8884e0ca-e537-474d-accd-3a698941907b-iptables-alerter-script\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-tmp-dir\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1abb1ad3-cc68-46d5-a07d-d2b4a3918de9-konnectivity-ca\") pod \"konnectivity-agent-mfjtk\" (UID: \"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9\") " pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-kubernetes\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-run\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8884e0ca-e537-474d-accd-3a698941907b-host-slash\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.974171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-hosts-file\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-sys\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysctl-d\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-etc-selinux\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8884e0ca-e537-474d-accd-3a698941907b-host-slash\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysctl-conf\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-sys\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-hosts-file\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.974443 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.974502 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:15:55.474481597 +0000 UTC m=+3.047102122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:54.974537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8884e0ca-e537-474d-accd-3a698941907b-iptables-alerter-script\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-kubernetes\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974651 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-lib-modules\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysconfig\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-systemd\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-tuned\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-etc-selinux\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-sysconfig\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmzr\" (UniqueName: \"kubernetes.io/projected/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-kube-api-access-zmmzr\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-systemd\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-device-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-device-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-host\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.974987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-sys-fs\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-sys-fs\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-host\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.975170 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-registration-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kd2\" (UniqueName: \"kubernetes.io/projected/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-kube-api-access-86kd2\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq86\" (UniqueName: \"kubernetes.io/projected/2d89f060-5022-4975-8fa5-2b17fb96675a-kube-api-access-6jq86\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975215 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1abb1ad3-cc68-46d5-a07d-d2b4a3918de9-konnectivity-ca\") pod \"konnectivity-agent-mfjtk\" (UID: \"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9\") " pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5g7\" (UniqueName: \"kubernetes.io/projected/8884e0ca-e537-474d-accd-3a698941907b-kube-api-access-nk5g7\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-var-lib-kubelet\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-socket-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d89f060-5022-4975-8fa5-2b17fb96675a-var-lib-kubelet\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975430 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-registration-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.976059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.975571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f96b8b8d-ba63-475d-9005-660e902fddff-socket-dir\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:54.977105 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.977079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1abb1ad3-cc68-46d5-a07d-d2b4a3918de9-agent-certs\") pod \"konnectivity-agent-mfjtk\" (UID: \"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9\") " pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:54.977583 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.977548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d89f060-5022-4975-8fa5-2b17fb96675a-tmp\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.977674 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.977607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d89f060-5022-4975-8fa5-2b17fb96675a-etc-tuned\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.986601 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.986576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq86\" (UniqueName: \"kubernetes.io/projected/2d89f060-5022-4975-8fa5-2b17fb96675a-kube-api-access-6jq86\") pod \"tuned-4xskk\" (UID: \"2d89f060-5022-4975-8fa5-2b17fb96675a\") " pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:54.990457 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.990438 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:15:54.990457 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.990458 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:15:54.990635 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.990468 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:54.990635 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:54.990528 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:15:55.490513292 +0000 UTC m=+3.063133830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:54.992838 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.992817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5g7\" (UniqueName: \"kubernetes.io/projected/8884e0ca-e537-474d-accd-3a698941907b-kube-api-access-nk5g7\") pod \"iptables-alerter-2jlnd\" (UID: \"8884e0ca-e537-474d-accd-3a698941907b\") " pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:54.993236 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.993212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmzr\" (UniqueName: \"kubernetes.io/projected/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-kube-api-access-zmmzr\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:54.994854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.994830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kd2\" (UniqueName: \"kubernetes.io/projected/36bf83e3-c725-41e9-b5b8-2b79ec57b7a7-kube-api-access-86kd2\") pod \"node-resolver-qc484\" (UID: \"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7\") " pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:54.995602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:54.995578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf6z4\" (UniqueName: \"kubernetes.io/projected/f96b8b8d-ba63-475d-9005-660e902fddff-kube-api-access-gf6z4\") pod \"aws-ebs-csi-driver-node-gq727\" (UID: \"f96b8b8d-ba63-475d-9005-660e902fddff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:55.050222 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.050185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" Jul 23 19:15:55.057972 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.057945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xn7q" Jul 23 19:15:55.065917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.065889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:15:55.072948 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.072923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdsw8" Jul 23 19:15:55.079947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.079919 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" Jul 23 19:15:55.086894 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.086871 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:15:55.093889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.093866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qc484" Jul 23 19:15:55.100889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.100854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xskk" Jul 23 19:15:55.104851 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.104824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2jlnd" Jul 23 19:15:55.172841 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.169566 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8884e0ca_e537_474d_accd_3a698941907b.slice/crio-2627e24ba347e496b902a25acb87b36bf424e7f596a2fa658b1a5aa15c8b82a6 WatchSource:0}: Error finding container 2627e24ba347e496b902a25acb87b36bf424e7f596a2fa658b1a5aa15c8b82a6: Status 404 returned error can't find the container with id 2627e24ba347e496b902a25acb87b36bf424e7f596a2fa658b1a5aa15c8b82a6 Jul 23 19:15:55.174215 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.174134 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4319b6_feb5_4089_8896_3e93d2e2e099.slice/crio-d21c74c35b1870ca97ac7a1cc8bbfbaaecd82416e658f25d3b681012129c65ed WatchSource:0}: Error finding container d21c74c35b1870ca97ac7a1cc8bbfbaaecd82416e658f25d3b681012129c65ed: Status 404 returned error can't find the container with id d21c74c35b1870ca97ac7a1cc8bbfbaaecd82416e658f25d3b681012129c65ed Jul 23 19:15:55.177974 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.177948 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96b8b8d_ba63_475d_9005_660e902fddff.slice/crio-78040cde66031ff93b5ac0bf6294a901bdca0317b56829df124bfbaf131fb4f5 WatchSource:0}: Error finding container 78040cde66031ff93b5ac0bf6294a901bdca0317b56829df124bfbaf131fb4f5: Status 404 returned error can't find the container with id 78040cde66031ff93b5ac0bf6294a901bdca0317b56829df124bfbaf131fb4f5 Jul 23 19:15:55.178673 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.178649 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b32df0e_06b0_4133_b367_edcf67395c90.slice/crio-442e41b1aaa2e10edd5bf491bbf59da6e8fb193deff4c88813a22239faebd18c WatchSource:0}: Error finding container 442e41b1aaa2e10edd5bf491bbf59da6e8fb193deff4c88813a22239faebd18c: Status 404 returned error can't find the container with id 442e41b1aaa2e10edd5bf491bbf59da6e8fb193deff4c88813a22239faebd18c Jul 23 19:15:55.179963 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.179933 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d89f060_5022_4975_8fa5_2b17fb96675a.slice/crio-284d89582c5ec6a65f8586099393ec0475a63aebe0342698f6ff7f74f9171cdc WatchSource:0}: Error finding container 284d89582c5ec6a65f8586099393ec0475a63aebe0342698f6ff7f74f9171cdc: Status 404 returned error can't find the container with id 284d89582c5ec6a65f8586099393ec0475a63aebe0342698f6ff7f74f9171cdc Jul 23 19:15:55.180987 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.180963 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da84ece_26a9_458f_841a_4b08e3060f97.slice/crio-69e5adfa7a4747d76e9a6c01e480632e56754c4d599bef6627111bd2e12f8c5c WatchSource:0}: Error finding container 69e5adfa7a4747d76e9a6c01e480632e56754c4d599bef6627111bd2e12f8c5c: Status 404 returned error can't find the container with id 69e5adfa7a4747d76e9a6c01e480632e56754c4d599bef6627111bd2e12f8c5c Jul 23 19:15:55.204249 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.204222 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abb1ad3_cc68_46d5_a07d_d2b4a3918de9.slice/crio-a749576c34d23e3cef0e5b702726db2b11f2d611aa5063a8de2ed470567f0b52 WatchSource:0}: Error finding container a749576c34d23e3cef0e5b702726db2b11f2d611aa5063a8de2ed470567f0b52: Status 404 returned error can't find the container with id a749576c34d23e3cef0e5b702726db2b11f2d611aa5063a8de2ed470567f0b52 Jul 23 19:15:55.205330 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:15:55.205260 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106b3961_d9dd_4575_8a01_01e11abaae6c.slice/crio-9f475f208332bca2b16d84c48de73f68958014aa84b8212b04ea9e96100af6e6 WatchSource:0}: Error finding container 9f475f208332bca2b16d84c48de73f68958014aa84b8212b04ea9e96100af6e6: Status 404 returned error can't find the container with id 9f475f208332bca2b16d84c48de73f68958014aa84b8212b04ea9e96100af6e6 Jul 23 19:15:55.377843 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.377660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:55.378017 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.377862 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:55.378017 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.377948 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:15:56.377924799 +0000 UTC m=+3.950545333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:55.478831 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.478533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:55.478831 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.478726 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:55.479051 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.478897 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:15:56.478879512 +0000 UTC m=+4.051500045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:55.580245 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.580200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:55.580448 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.580393 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:15:55.580448 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.580417 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:15:55.580448 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.580432 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:55.580623 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.580507 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:15:56.580484398 +0000 UTC m=+4.153104943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:55.804102 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.803891 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 19:10:55 +0000 UTC" deadline="2028-02-22 01:59:58.326945333 +0000 UTC" Jul 23 19:15:55.804102 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.803953 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13878h44m2.522997307s" Jul 23 19:15:55.841751 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.841204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:55.841751 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:55.841346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:15:55.875025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.874062 2574 generic.go:356] "Generic (PLEG): container finished" podID="802635c36462f36fb851b8a8323d592f" containerID="b1468befa884d31eb3362b1ec46837c1524eb1536c902952ee1584654e06b317" exitCode=0 Jul 23 19:15:55.875025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.874779 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" event={"ID":"802635c36462f36fb851b8a8323d592f","Type":"ContainerDied","Data":"b1468befa884d31eb3362b1ec46837c1524eb1536c902952ee1584654e06b317"} Jul 23 19:15:55.891471 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.891371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mfjtk" event={"ID":"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9","Type":"ContainerStarted","Data":"a749576c34d23e3cef0e5b702726db2b11f2d611aa5063a8de2ed470567f0b52"} Jul 23 19:15:55.907036 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.907000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerStarted","Data":"442e41b1aaa2e10edd5bf491bbf59da6e8fb193deff4c88813a22239faebd18c"} Jul 23 19:15:55.917229 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.917194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" event={"ID":"f96b8b8d-ba63-475d-9005-660e902fddff","Type":"ContainerStarted","Data":"78040cde66031ff93b5ac0bf6294a901bdca0317b56829df124bfbaf131fb4f5"} Jul 23 19:15:55.932079 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.932039 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xn7q" event={"ID":"8d4319b6-feb5-4089-8896-3e93d2e2e099","Type":"ContainerStarted","Data":"d21c74c35b1870ca97ac7a1cc8bbfbaaecd82416e658f25d3b681012129c65ed"} Jul 23 19:15:55.940133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.940100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qc484" event={"ID":"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7","Type":"ContainerStarted","Data":"6202a02fa5f8f8151fc9ef3ef949f44ab44406e0856ea2265e3320c19567dd98"} Jul 23 19:15:55.948112 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.948073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdsw8" event={"ID":"106b3961-d9dd-4575-8a01-01e11abaae6c","Type":"ContainerStarted","Data":"9f475f208332bca2b16d84c48de73f68958014aa84b8212b04ea9e96100af6e6"} Jul 23 19:15:55.953482 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.953449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"69e5adfa7a4747d76e9a6c01e480632e56754c4d599bef6627111bd2e12f8c5c"} Jul 23 19:15:55.957060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.957029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xskk" event={"ID":"2d89f060-5022-4975-8fa5-2b17fb96675a","Type":"ContainerStarted","Data":"284d89582c5ec6a65f8586099393ec0475a63aebe0342698f6ff7f74f9171cdc"} Jul 23 19:15:55.981562 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:55.981521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2jlnd" event={"ID":"8884e0ca-e537-474d-accd-3a698941907b","Type":"ContainerStarted","Data":"2627e24ba347e496b902a25acb87b36bf424e7f596a2fa658b1a5aa15c8b82a6"} Jul 23 19:15:56.389322 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.388096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:56.389322 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.388242 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:56.389322 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.388317 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:15:58.388297596 +0000 UTC m=+5.960918119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:56.489554 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.489511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:56.489733 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.489697 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:56.489834 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.489771 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:15:58.489747533 +0000 UTC m=+6.062368067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:56.590598 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.590558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:56.590947 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.590758 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:15:56.590947 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.590804 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:15:56.590947 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.590818 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:56.590947 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.590881 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:15:58.590860531 +0000 UTC m=+6.163481073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:56.804355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.804258 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 19:10:55 +0000 UTC" deadline="2028-04-09 06:05:22.829810658 +0000 UTC" Jul 23 19:15:56.804355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.804301 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15010h49m26.025514263s" Jul 23 19:15:56.841787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.840839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:56.841787 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.841001 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:15:56.841787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.841566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:56.841787 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:56.841678 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:15:56.855172 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.854366 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 19:15:56.988873 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:56.988833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" event={"ID":"802635c36462f36fb851b8a8323d592f","Type":"ContainerStarted","Data":"570110cfb0101fe05d4af410812adb9dcc98859f4b853dda3929cdb74d81261e"} Jul 23 19:15:57.840729 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:57.840688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:57.841255 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:57.840865 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:15:58.409648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:58.409210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:58.409648 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.409414 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:58.409648 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.409486 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:02.409466315 +0000 UTC m=+9.982086851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:15:58.510380 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:58.510339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:58.510589 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.510571 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:58.510663 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.510647 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:02.510626949 +0000 UTC m=+10.083247485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:15:58.611602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:58.611553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:58.611810 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.611768 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:15:58.611891 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.611811 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:15:58.611891 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.611826 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:58.612005 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.611897 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:16:02.611874263 +0000 UTC m=+10.184494828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:15:58.845857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:58.845738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:15:58.846311 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.845892 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:15:58.846311 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:58.846101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:15:58.846311 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.846218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:15:58.846311 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:15:58.846286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:15:58.846557 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:15:58.846395 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:00.841074 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:00.840994 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:00.841555 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:00.841131 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:00.841555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:00.841146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:00.841555 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:00.841233 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:00.841555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:00.841282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:00.841555 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:00.841400 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:02.449433 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:02.449389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:02.449899 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.449561 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:16:02.449899 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.449627 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:10.449609407 +0000 UTC m=+18.022229926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:16:02.550518 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:02.550475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:02.550697 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.550683 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:16:02.550791 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.550779 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:10.550755601 +0000 UTC m=+18.123376134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:16:02.651313 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:02.651273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:02.651541 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.651465 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:16:02.651541 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.651494 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:16:02.651541 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.651509 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:16:02.651725 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.651575 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:16:10.651555725 +0000 UTC m=+18.224176259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:16:02.844970 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:02.844830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:02.844970 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:02.844858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:02.845173 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.844964 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:02.845173 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.845020 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:02.845173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:02.845085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:02.845173 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:02.845166 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:04.840763 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:04.840667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:04.840763 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:04.840671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:04.841285 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:04.840797 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:04.841285 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:04.840879 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:04.841285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:04.840673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:04.841285 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:04.840980 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:06.840797 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:06.840755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:06.840797 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:06.840796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:06.841332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:06.840761 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:06.841332 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:06.840899 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:06.841332 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:06.841023 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:06.841332 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:06.841095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:08.841121 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:08.841084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:08.841584 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:08.841086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:08.841584 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:08.841223 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:08.841584 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:08.841086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:08.841584 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:08.841264 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:08.841584 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:08.841334 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:10.511412 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:10.511367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:10.511836 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.511517 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:16:10.511836 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.511586 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:26.511570465 +0000 UTC m=+34.084190982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:16:10.612155 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:10.612112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:10.612306 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.612275 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:16:10.612348 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.612340 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:26.612324941 +0000 UTC m=+34.184945458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:16:10.713184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:10.713143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:10.713342 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.713319 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:16:10.713401 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.713349 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:16:10.713401 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.713360 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:16:10.713471 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.713410 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:16:26.71339823 +0000 UTC m=+34.286018748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:16:10.841075 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:10.840801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:10.841075 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:10.840877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:10.841075 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:10.840959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:10.841075 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.840972 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:10.841075 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.841074 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:10.841461 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:10.841159 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:12.841767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:12.841714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:12.842202 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:12.841834 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:12.842202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:12.841847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:12.842202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:12.841866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:12.842202 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:12.841978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:12.842202 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:12.842066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:14.850353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:14.848026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:14.850353 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:14.848158 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:14.850353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:14.848543 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:14.850353 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:14.848623 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:14.850353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:14.848677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:14.850353 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:14.848762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:15.040877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.040604 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" event={"ID":"ade0de55707084dd778646963520c9b6","Type":"ContainerStarted","Data":"5b034d1a8d264335a8815a813ee45ec094bbf7b5c1f5c400f990ecde9f494f75"} Jul 23 19:16:15.042866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.042828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdsw8" event={"ID":"106b3961-d9dd-4575-8a01-01e11abaae6c","Type":"ContainerStarted","Data":"8e3a06aa179a22e823254a9a09a58f99d399dc72c9ece254b345c4495953d2af"} Jul 23 19:16:15.045079 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.045056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"c6e04d2020d064be52251c9c0091516a3cc0d59294f9189799e6912a5a81f767"} Jul 23 19:16:15.046689 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.046662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xskk" event={"ID":"2d89f060-5022-4975-8fa5-2b17fb96675a","Type":"ContainerStarted","Data":"e900024daf85c971ac90b87e80ef89d2a0eeefd99ac9e27de2e0871b7b31d1d5"} Jul 23 19:16:15.048253 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.048200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mfjtk" event={"ID":"1abb1ad3-cc68-46d5-a07d-d2b4a3918de9","Type":"ContainerStarted","Data":"a9c380a85e3bbbcee7a4ee4c9798e2f2709c4f3a4480e1d18ead1dedb86b5985"} Jul 23 19:16:15.050002 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.049980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerStarted","Data":"2dd242fd9b664793dbca9d2e0d37ee6cda45022902bab1e29a49988c9184b557"} Jul 23 19:16:15.051716 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.051656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" event={"ID":"f96b8b8d-ba63-475d-9005-660e902fddff","Type":"ContainerStarted","Data":"b497288f4342d6b017c9282d38957b39a7ce5593aecbe03b91e098efa196da3a"} Jul 23 19:16:15.053525 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.053504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xn7q" event={"ID":"8d4319b6-feb5-4089-8896-3e93d2e2e099","Type":"ContainerStarted","Data":"fda2678634c165cb8d49bc5d941df040937961d5e02b62284eaf1d9cb68fb0b1"} Jul 23 19:16:15.054972 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.054932 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qc484" event={"ID":"36bf83e3-c725-41e9-b5b8-2b79ec57b7a7","Type":"ContainerStarted","Data":"aaa3d853bd58755d1097972f414e53f1ee7659c5101a8819075b28c53bc984d2"} Jul 23 19:16:15.059012 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.058611 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-134.ec2.internal" podStartSLOduration=21.058592358 podStartE2EDuration="21.058592358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:15:57.009498744 +0000 UTC m=+4.582119291" watchObservedRunningTime="2026-07-23 19:16:15.058592358 +0000 UTC m=+22.631212900" Jul 23 19:16:15.059012 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.058936 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-134.ec2.internal" podStartSLOduration=21.058928203 podStartE2EDuration="21.058928203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:15.058238824 +0000 UTC m=+22.630859365" watchObservedRunningTime="2026-07-23 19:16:15.058928203 +0000 UTC m=+22.631548745" Jul 23 19:16:15.075723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.075484 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4xskk" podStartSLOduration=2.465108226 podStartE2EDuration="22.075463441s" totalImagesPullingTime="19.610355215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:15.074358229 +0000 UTC m=+22.646978767" watchObservedRunningTime="2026-07-23 19:16:15.075463441 +0000 UTC m=+22.648083981" Jul 23 19:16:15.119749 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.119556 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4xn7q" podStartSLOduration=2.501959461 podStartE2EDuration="22.119539547s" totalImagesPullingTime="19.617580086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:15.119282363 +0000 UTC m=+22.691902902" watchObservedRunningTime="2026-07-23 19:16:15.119539547 +0000 UTC m=+22.692160087" Jul 23 19:16:15.141507 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.141455 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vdsw8" podStartSLOduration=2.494587703 podStartE2EDuration="22.141434235s" totalImagesPullingTime="19.646846532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:15.140586817 +0000 UTC m=+22.713207360" watchObservedRunningTime="2026-07-23 19:16:15.141434235 +0000 UTC m=+22.714054775" Jul 23 19:16:15.165790 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.165720 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mfjtk" podStartSLOduration=2.578011079 podStartE2EDuration="22.165702077s" totalImagesPullingTime="19.587690998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:15.165590055 +0000 UTC m=+22.738210588" watchObservedRunningTime="2026-07-23 19:16:15.165702077 +0000 UTC m=+22.738322620" Jul 23 19:16:15.187607 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:15.187564 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qc484" podStartSLOduration=2.5666558139999998 podStartE2EDuration="22.187548539s" totalImagesPullingTime="19.620892725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:15.187021777 +0000 UTC m=+22.759642318" watchObservedRunningTime="2026-07-23 19:16:15.187548539 +0000 UTC m=+22.760169077" Jul 23 19:16:16.058809 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.058773 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b32df0e-06b0-4133-b367-edcf67395c90" containerID="2dd242fd9b664793dbca9d2e0d37ee6cda45022902bab1e29a49988c9184b557" exitCode=0 Jul 23 19:16:16.059592 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.058862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerDied","Data":"2dd242fd9b664793dbca9d2e0d37ee6cda45022902bab1e29a49988c9184b557"} Jul 23 19:16:16.061982 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.061950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"8c5a5a4b15f36ee31377c41a25f5592867ae22ebc6e4f26471f1da9c01e06677"} Jul 23 19:16:16.062104 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.062041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"3a5523720652a08f467a2d1d98c8b4ef91acb58539720beb5a8aa6c5b129e3dc"} Jul 23 19:16:16.062104 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.062077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"e71c7d5e4fa41d1d81a5e85bffc2a54a9afdfd154f08a3d36b4840c3e4214a30"} Jul 23 19:16:16.062179 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.062113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"49eb6852cff846d4d9e738e4440310cb15c4c77ddbc4cc6273de86cb94ec1e6e"} Jul 23 19:16:16.062179 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.062147 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"123a381e64b294e76e387d08d494bf6bdbe0fc10cceba544ca759210b099f7ab"} Jul 23 19:16:16.248134 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.248113 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 19:16:16.824843 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.824651 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T19:16:16.248130157Z","UUID":"0d4f7585-f3ec-4758-9da7-b0dd4c2a42a8","Handler":null,"Name":"","Endpoint":""} Jul 23 19:16:16.827686 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.827663 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 19:16:16.827837 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.827711 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 19:16:16.841481 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.841448 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:16.841656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.841449 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:16.841656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:16.841594 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:16.841656 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:16.841593 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:16.841842 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:16.841706 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:16.841899 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:16.841835 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:17.067430 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:17.067390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" event={"ID":"f96b8b8d-ba63-475d-9005-660e902fddff","Type":"ContainerStarted","Data":"c96ff1afd168840c031ab95f2bda95666479bb34fb2c0f5b90a0541550e0bae2"} Jul 23 19:16:17.069266 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:17.069206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2jlnd" event={"ID":"8884e0ca-e537-474d-accd-3a698941907b","Type":"ContainerStarted","Data":"6ae86a68b2b40f0f8a7aa34d6714cc1a2d2ce3ba3b348836b607b84fb7ccfddf"} Jul 23 19:16:18.074193 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:18.074069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" event={"ID":"f96b8b8d-ba63-475d-9005-660e902fddff","Type":"ContainerStarted","Data":"f10668129386d7fcce08ceb090864aaaf4549cd6398f563aa8fc87633a4190f5"} Jul 23 19:16:18.096009 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:18.095962 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gq727" podStartSLOduration=3.226241816 podStartE2EDuration="25.095942436s" totalImagesPullingTime="21.86970062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:18.095840556 +0000 UTC m=+25.668461095" watchObservedRunningTime="2026-07-23 19:16:18.095942436 +0000 UTC m=+25.668562976" Jul 23 19:16:18.096281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:18.096251 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2jlnd" podStartSLOduration=5.476141345 podStartE2EDuration="25.096242359s" totalImagesPullingTime="19.620101014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:17.090983956 +0000 UTC m=+24.663604495" watchObservedRunningTime="2026-07-23 19:16:18.096242359 +0000 UTC m=+25.668862900" Jul 23 19:16:18.841393 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:18.841197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:18.841568 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:18.841197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:18.841568 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:18.841491 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:18.841568 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:18.841197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:18.841568 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:18.841551 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:18.841774 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:18.841609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:19.080283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:19.080242 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"27e8ee1e50ae3f6965c5cc1a9ccd23b77da5e90b26c0691f0e5bce607266f89c"} Jul 23 19:16:20.041239 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.041198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:16:20.041862 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.041841 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:16:20.082386 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.082356 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:16:20.082966 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.082948 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mfjtk" Jul 23 19:16:20.841560 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.841393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:20.841685 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.841393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:20.841685 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:20.841647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:20.841782 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:20.841440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:20.841782 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:20.841712 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:20.841850 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:20.841814 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:21.091035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:21.090918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" event={"ID":"9da84ece-26a9-458f-841a-4b08e3060f97","Type":"ContainerStarted","Data":"b603ddbdf43fe03fc9ef0d85a9c7c757ed3e443f5a5e8b5830a56fe257e87b9e"} Jul 23 19:16:21.091898 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:21.091221 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:16:21.092797 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:21.092775 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b32df0e-06b0-4133-b367-edcf67395c90" containerID="94312fc87997c5f82734d3b8df2b25a2eb765ce2929410a8d86d2650d60416e0" exitCode=0 Jul 23 19:16:21.092912 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:21.092854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerDied","Data":"94312fc87997c5f82734d3b8df2b25a2eb765ce2929410a8d86d2650d60416e0"} Jul 23 19:16:21.106668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:21.106645 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:16:21.162030 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:21.161968 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" podStartSLOduration=8.497913068999999 podStartE2EDuration="28.161952929s" totalImagesPullingTime="19.66403986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:21.127561757 +0000 UTC m=+28.700182308" watchObservedRunningTime="2026-07-23 19:16:21.161952929 +0000 UTC m=+28.734573468" Jul 23 19:16:22.095329 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.095293 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:16:22.095912 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.095680 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:16:22.109404 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.109382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:16:22.734434 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.734401 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2gqgt"] Jul 23 19:16:22.734600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.734563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:22.734671 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:22.734650 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:22.737660 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.737635 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8679f"] Jul 23 19:16:22.737816 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.737776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:22.737882 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:22.737865 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:22.749381 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.749351 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5szb"] Jul 23 19:16:22.749556 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:22.749472 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:22.749621 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:22.749580 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:23.099507 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:23.099421 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b32df0e-06b0-4133-b367-edcf67395c90" containerID="b8c0fbd890170fd4521a74e21a471da1ceaee55e755f7cced7835062772d5584" exitCode=0 Jul 23 19:16:23.100075 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:23.099512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerDied","Data":"b8c0fbd890170fd4521a74e21a471da1ceaee55e755f7cced7835062772d5584"} Jul 23 19:16:24.840971 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:24.840937 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:24.841351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:24.840937 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:24.841351 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:24.841056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:24.841351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:24.840941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:24.841351 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:24.841163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:24.841351 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:24.841218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:25.106990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:25.106911 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b32df0e-06b0-4133-b367-edcf67395c90" containerID="faf42eaa8bc5011de720ef76783c9d2391e8695d105a7528ad139d7ac7daa260" exitCode=0 Jul 23 19:16:25.106990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:25.106969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerDied","Data":"faf42eaa8bc5011de720ef76783c9d2391e8695d105a7528ad139d7ac7daa260"} Jul 23 19:16:26.531278 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:26.531021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:26.531701 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.531210 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 19:16:26.531701 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.531364 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret podName:6b0171d6-8abc-4a3e-a72a-f29271153a70 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:58.531342678 +0000 UTC m=+66.103963216 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret") pod "global-pull-secret-syncer-2gqgt" (UID: "6b0171d6-8abc-4a3e-a72a-f29271153a70") : object "kube-system"/"original-pull-secret" not registered Jul 23 19:16:26.632554 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:26.632464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:26.632713 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.632621 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:16:26.632713 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.632691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs podName:3ab9bc48-2193-4381-848e-fb7f5ba5fca0 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:58.632674655 +0000 UTC m=+66.205295173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs") pod "network-metrics-daemon-8679f" (UID: "3ab9bc48-2193-4381-848e-fb7f5ba5fca0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 19:16:26.732986 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:26.732954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:26.733165 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.733132 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 19:16:26.733165 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.733159 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 19:16:26.733280 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.733172 2574 projected.go:196] Error preparing data for projected volume kube-api-access-x78fd for pod openshift-network-diagnostics/network-check-target-x5szb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:16:26.733280 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.733235 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd podName:5eefc014-bc40-4c10-8246-6f7ae61cfa1b nodeName:}" failed. No retries permitted until 2026-07-23 19:16:58.733216291 +0000 UTC m=+66.305836819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x78fd" (UniqueName: "kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd") pod "network-check-target-x5szb" (UID: "5eefc014-bc40-4c10-8246-6f7ae61cfa1b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 19:16:26.841397 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:26.841362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:26.841585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:26.841362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:26.841585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:26.841513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:26.841585 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.841509 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8679f" podUID="3ab9bc48-2193-4381-848e-fb7f5ba5fca0" Jul 23 19:16:26.841762 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.841669 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2gqgt" podUID="6b0171d6-8abc-4a3e-a72a-f29271153a70" Jul 23 19:16:26.841827 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:26.841778 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5szb" podUID="5eefc014-bc40-4c10-8246-6f7ae61cfa1b" Jul 23 19:16:28.732034 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.732004 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-134.ec2.internal" event="NodeReady" Jul 23 19:16:28.732462 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.732158 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 19:16:28.787327 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.787292 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-578c5786db-p5fc7"] Jul 23 19:16:28.790782 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.790754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.794287 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.794259 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 19:16:28.794881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.794856 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p6k4c\"" Jul 23 19:16:28.795247 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.795226 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 19:16:28.795516 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.795493 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 19:16:28.801343 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.801313 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n4v72"] Jul 23 19:16:28.801756 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.801722 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 19:16:28.804146 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.804128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:28.808237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.808190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-578c5786db-p5fc7"] Jul 23 19:16:28.808887 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.808866 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 19:16:28.808987 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.808872 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gvmps\"" Jul 23 19:16:28.808987 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.808925 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 19:16:28.813870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.813634 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hj9rw"] Jul 23 19:16:28.816618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.816596 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:28.819358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.819336 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 19:16:28.819447 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.819361 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8cbrx\"" Jul 23 19:16:28.820990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.820964 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 19:16:28.821083 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.820975 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 19:16:28.823378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.823342 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n4v72"] Jul 23 19:16:28.832399 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.832371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hj9rw"] Jul 23 19:16:28.844787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.844760 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:28.844937 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.844796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:28.844937 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.844856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:28.847638 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.847603 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 19:16:28.847786 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.847710 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v5npk\"" Jul 23 19:16:28.847786 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.847745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 19:16:28.847899 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.847824 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 19:16:28.848064 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.848046 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6wbwl\"" Jul 23 19:16:28.848219 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.848179 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 19:16:28.949639 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zzmt\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-kube-api-access-9zzmt\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.949828 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcln6\" (UniqueName: \"kubernetes.io/projected/21caab7f-0bf4-44bd-8a75-21db01239fb5-kube-api-access-kcln6\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:28.949828 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-trusted-ca\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.949828 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-image-registry-private-configuration\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.949828 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e2c84f-1b54-4bed-8c78-5ebf0253c790-ca-trust-extracted\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.950025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-bound-sa-token\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.950025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f25159-b9cc-4653-8ed2-d54b0a01f3e0-cert\") pod \"ingress-canary-hj9rw\" (UID: \"24f25159-b9cc-4653-8ed2-d54b0a01f3e0\") " pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:28.950025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21caab7f-0bf4-44bd-8a75-21db01239fb5-tmp-dir\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:28.950025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-certificates\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.950025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.949999 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-installation-pull-secrets\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.950025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.950019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21caab7f-0bf4-44bd-8a75-21db01239fb5-config-volume\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:28.950252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.950056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:28.950252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.950074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpg2\" (UniqueName: \"kubernetes.io/projected/24f25159-b9cc-4653-8ed2-d54b0a01f3e0-kube-api-access-hwpg2\") pod \"ingress-canary-hj9rw\" (UID: \"24f25159-b9cc-4653-8ed2-d54b0a01f3e0\") " pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:28.950252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.950088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:28.973499 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:28.973463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mfjtk_1abb1ad3-cc68-46d5-a07d-d2b4a3918de9/konnectivity-agent/0.log" limit=1024 Jul 23 19:16:29.051541 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-bound-sa-token\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.051541 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f25159-b9cc-4653-8ed2-d54b0a01f3e0-cert\") pod \"ingress-canary-hj9rw\" (UID: \"24f25159-b9cc-4653-8ed2-d54b0a01f3e0\") " pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:29.051541 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21caab7f-0bf4-44bd-8a75-21db01239fb5-tmp-dir\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-certificates\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-installation-pull-secrets\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21caab7f-0bf4-44bd-8a75-21db01239fb5-config-volume\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpg2\" (UniqueName: \"kubernetes.io/projected/24f25159-b9cc-4653-8ed2-d54b0a01f3e0-kube-api-access-hwpg2\") pod \"ingress-canary-hj9rw\" (UID: \"24f25159-b9cc-4653-8ed2-d54b0a01f3e0\") " pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zzmt\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-kube-api-access-9zzmt\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.051783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcln6\" (UniqueName: \"kubernetes.io/projected/21caab7f-0bf4-44bd-8a75-21db01239fb5-kube-api-access-kcln6\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.052050 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-trusted-ca\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.052050 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.051902 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:16:29.052050 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.051923 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578c5786db-p5fc7: secret "image-registry-tls" not found Jul 23 19:16:29.052050 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.051933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21caab7f-0bf4-44bd-8a75-21db01239fb5-tmp-dir\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.052050 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.051997 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls podName:22e2c84f-1b54-4bed-8c78-5ebf0253c790 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:29.551974535 +0000 UTC m=+37.124595056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls") pod "image-registry-578c5786db-p5fc7" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790") : secret "image-registry-tls" not found Jul 23 19:16:29.052050 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.052009 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:16:29.052260 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.052058 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls podName:21caab7f-0bf4-44bd-8a75-21db01239fb5 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:29.552043789 +0000 UTC m=+37.124664326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls") pod "dns-default-n4v72" (UID: "21caab7f-0bf4-44bd-8a75-21db01239fb5") : secret "dns-default-metrics-tls" not found Jul 23 19:16:29.052260 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.052203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-image-registry-private-configuration\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.052260 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.052243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e2c84f-1b54-4bed-8c78-5ebf0253c790-ca-trust-extracted\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.053046 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.052679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e2c84f-1b54-4bed-8c78-5ebf0253c790-ca-trust-extracted\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.053046 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.053001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21caab7f-0bf4-44bd-8a75-21db01239fb5-config-volume\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.053222 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.053146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-certificates\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.053564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.053544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-trusted-ca\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.056343 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.056283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-image-registry-private-configuration\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.056343 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.056298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24f25159-b9cc-4653-8ed2-d54b0a01f3e0-cert\") pod \"ingress-canary-hj9rw\" (UID: \"24f25159-b9cc-4653-8ed2-d54b0a01f3e0\") " pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:29.056343 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.056286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-installation-pull-secrets\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.066132 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.066100 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-bound-sa-token\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.076986 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.076952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zzmt\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-kube-api-access-9zzmt\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.076986 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.076979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpg2\" (UniqueName: \"kubernetes.io/projected/24f25159-b9cc-4653-8ed2-d54b0a01f3e0-kube-api-access-hwpg2\") pod \"ingress-canary-hj9rw\" (UID: \"24f25159-b9cc-4653-8ed2-d54b0a01f3e0\") " pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:29.077335 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.077314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcln6\" (UniqueName: \"kubernetes.io/projected/21caab7f-0bf4-44bd-8a75-21db01239fb5-kube-api-access-kcln6\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.086879 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.086844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mfjtk_1abb1ad3-cc68-46d5-a07d-d2b4a3918de9/konnectivity-agent/0.log" limit=1024 Jul 23 19:16:29.127647 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.127616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hj9rw" Jul 23 19:16:29.474285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.474105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hj9rw"] Jul 23 19:16:29.477226 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:29.477201 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f25159_b9cc_4653_8ed2_d54b0a01f3e0.slice/crio-0a2ba7d3b55ddb3ccb08900f35c12c775c19a1af0f4fcb9756318afd7218876e WatchSource:0}: Error finding container 0a2ba7d3b55ddb3ccb08900f35c12c775c19a1af0f4fcb9756318afd7218876e: Status 404 returned error can't find the container with id 0a2ba7d3b55ddb3ccb08900f35c12c775c19a1af0f4fcb9756318afd7218876e Jul 23 19:16:29.556627 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.556592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:29.556799 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:29.556636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:29.556799 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.556774 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:16:29.556799 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.556794 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578c5786db-p5fc7: secret "image-registry-tls" not found Jul 23 19:16:29.557013 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.556812 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:16:29.557013 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.556848 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls podName:22e2c84f-1b54-4bed-8c78-5ebf0253c790 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:30.556833772 +0000 UTC m=+38.129454312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls") pod "image-registry-578c5786db-p5fc7" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790") : secret "image-registry-tls" not found Jul 23 19:16:29.557013 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:29.556861 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls podName:21caab7f-0bf4-44bd-8a75-21db01239fb5 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:30.556855401 +0000 UTC m=+38.129475919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls") pod "dns-default-n4v72" (UID: "21caab7f-0bf4-44bd-8a75-21db01239fb5") : secret "dns-default-metrics-tls" not found Jul 23 19:16:30.120344 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:30.120309 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hj9rw" event={"ID":"24f25159-b9cc-4653-8ed2-d54b0a01f3e0","Type":"ContainerStarted","Data":"0a2ba7d3b55ddb3ccb08900f35c12c775c19a1af0f4fcb9756318afd7218876e"} Jul 23 19:16:30.123283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:30.123257 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b32df0e-06b0-4133-b367-edcf67395c90" containerID="414b819d486113e7f98584f89a4b52f4c9274388eabca0ba1c0e2597d81dc84d" exitCode=0 Jul 23 19:16:30.123403 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:30.123304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerDied","Data":"414b819d486113e7f98584f89a4b52f4c9274388eabca0ba1c0e2597d81dc84d"} Jul 23 19:16:30.564016 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:30.563985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:30.564220 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:30.564026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:30.564220 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:30.564173 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:16:30.564327 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:30.564236 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls podName:21caab7f-0bf4-44bd-8a75-21db01239fb5 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:32.56421664 +0000 UTC m=+40.136837165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls") pod "dns-default-n4v72" (UID: "21caab7f-0bf4-44bd-8a75-21db01239fb5") : secret "dns-default-metrics-tls" not found Jul 23 19:16:30.564506 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:30.564483 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:16:30.564506 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:30.564505 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578c5786db-p5fc7: secret "image-registry-tls" not found Jul 23 19:16:30.564698 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:30.564569 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls podName:22e2c84f-1b54-4bed-8c78-5ebf0253c790 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:32.564553317 +0000 UTC m=+40.137173845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls") pod "image-registry-578c5786db-p5fc7" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790") : secret "image-registry-tls" not found Jul 23 19:16:31.130144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:31.130107 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b32df0e-06b0-4133-b367-edcf67395c90" containerID="4871259916c77dc7d4b8d21f922d6c17e76ca7362b8b5e684b7067a6e49da843" exitCode=0 Jul 23 19:16:31.130596 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:31.130200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerDied","Data":"4871259916c77dc7d4b8d21f922d6c17e76ca7362b8b5e684b7067a6e49da843"} Jul 23 19:16:32.136751 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:32.136703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" event={"ID":"9b32df0e-06b0-4133-b367-edcf67395c90","Type":"ContainerStarted","Data":"9621bab7522ce330bf98b68a2d497a6a32f7fda817ed010efd018788cbb1fd96"} Jul 23 19:16:32.167050 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:32.166616 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4w5f5" podStartSLOduration=13.342410855 podStartE2EDuration="39.166592412s" totalImagesPullingTime="25.824181557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:32.165355308 +0000 UTC m=+39.737975849" watchObservedRunningTime="2026-07-23 19:16:32.166592412 +0000 UTC m=+39.739212953" Jul 23 19:16:32.578564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:32.578514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:32.578564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:32.578570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:32.578821 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:32.578698 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:16:32.578821 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:32.578724 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578c5786db-p5fc7: secret "image-registry-tls" not found Jul 23 19:16:32.578821 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:32.578781 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:16:32.578821 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:32.578811 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls podName:22e2c84f-1b54-4bed-8c78-5ebf0253c790 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:36.578790135 +0000 UTC m=+44.151410666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls") pod "image-registry-578c5786db-p5fc7" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790") : secret "image-registry-tls" not found Jul 23 19:16:32.579007 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:32.578837 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls podName:21caab7f-0bf4-44bd-8a75-21db01239fb5 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:36.578821316 +0000 UTC m=+44.151441858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls") pod "dns-default-n4v72" (UID: "21caab7f-0bf4-44bd-8a75-21db01239fb5") : secret "dns-default-metrics-tls" not found Jul 23 19:16:33.140441 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.140349 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hj9rw" event={"ID":"24f25159-b9cc-4653-8ed2-d54b0a01f3e0","Type":"ContainerStarted","Data":"23d7685c2239e9f112c006ad0d7a4ba393b340c4ec135a3001b78eba7c699538"} Jul 23 19:16:33.157916 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.157874 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hj9rw" podStartSLOduration=1.843893412 podStartE2EDuration="5.157859906s" totalImagesPullingTime="3.313966494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:33.156778602 +0000 UTC m=+40.729399142" watchObservedRunningTime="2026-07-23 19:16:33.157859906 +0000 UTC m=+40.730480450" Jul 23 19:16:33.359620 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.359588 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq"] Jul 23 19:16:33.380443 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.380412 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw"] Jul 23 19:16:33.380613 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.380522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" Jul 23 19:16:33.384441 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.384403 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.384599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.384445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g5m\" (UniqueName: \"kubernetes.io/projected/6e34a4ee-f7cc-4171-ad03-75227ccbc291-kube-api-access-n8g5m\") pod \"volume-data-source-validator-78cf4c6f6-hzgwq\" (UID: \"6e34a4ee-f7cc-4171-ad03-75227ccbc291\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" Jul 23 19:16:33.384599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.384417 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-t9k7z\"" Jul 23 19:16:33.385004 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.384984 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.404519 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.404441 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64cbdc95dd-q7nph"] Jul 23 19:16:33.404661 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.404576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.407268 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.407236 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 19:16:33.407369 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.407273 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.407695 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.407673 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 19:16:33.407830 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.407755 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.410157 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.410080 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-c2qc4\"" Jul 23 19:16:33.420435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.420413 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq"] Jul 23 19:16:33.420521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.420452 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw"] Jul 23 19:16:33.420521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.420473 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64cbdc95dd-q7nph"] Jul 23 19:16:33.420596 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.420518 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.423514 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.423497 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 19:16:33.423616 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.423523 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 19:16:33.423616 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.423523 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 19:16:33.423845 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.423831 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.423929 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.423832 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 19:16:33.423985 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.423944 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.424172 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.424153 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-b6gw9\"" Jul 23 19:16:33.469406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.469374 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf"] Jul 23 19:16:33.485641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.485814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-default-certificate\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.485814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-stats-auth\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.485814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8b21220-6461-4e91-9760-27f3914bf6d4-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.485947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.485947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttxh\" (UniqueName: \"kubernetes.io/projected/10489313-35b1-47b5-a9b0-1e2df4a66bf6-kube-api-access-9ttxh\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.485947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485848 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6n2\" (UniqueName: \"kubernetes.io/projected/a8b21220-6461-4e91-9760-27f3914bf6d4-kube-api-access-pz6n2\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.485947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8b21220-6461-4e91-9760-27f3914bf6d4-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.486061 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.485949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g5m\" (UniqueName: \"kubernetes.io/projected/6e34a4ee-f7cc-4171-ad03-75227ccbc291-kube-api-access-n8g5m\") pod \"volume-data-source-validator-78cf4c6f6-hzgwq\" (UID: \"6e34a4ee-f7cc-4171-ad03-75227ccbc291\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" Jul 23 19:16:33.492507 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.492479 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5zzxw"] Jul 23 19:16:33.492637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.492619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" Jul 23 19:16:33.497381 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.497361 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dvbht\"" Jul 23 19:16:33.502480 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.502456 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cr79v"] Jul 23 19:16:33.502572 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.502554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.505824 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.505803 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 19:16:33.505957 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.505840 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.506189 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.506157 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.506328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.506314 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 19:16:33.506426 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.506402 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-48rrm\"" Jul 23 19:16:33.514682 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.514660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf"] Jul 23 19:16:33.514836 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.514774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.518537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.518345 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 19:16:33.519954 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.519931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cr79v"] Jul 23 19:16:33.520367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.520345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g5m\" (UniqueName: \"kubernetes.io/projected/6e34a4ee-f7cc-4171-ad03-75227ccbc291-kube-api-access-n8g5m\") pod \"volume-data-source-validator-78cf4c6f6-hzgwq\" (UID: \"6e34a4ee-f7cc-4171-ad03-75227ccbc291\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" Jul 23 19:16:33.520961 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.520944 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5zzxw"] Jul 23 19:16:33.522152 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.522139 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 19:16:33.522866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.522851 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.522950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.522922 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ztm4w\"" Jul 23 19:16:33.523393 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.523378 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.523446 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.523382 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 19:16:33.532289 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.532268 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 19:16:33.536482 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.536461 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx"] Jul 23 19:16:33.552933 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.552909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.555222 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.555195 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 19:16:33.555329 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.555238 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.555329 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.555258 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nvxnb\"" Jul 23 19:16:33.555669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.555632 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.555807 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.555698 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 19:16:33.556776 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.556757 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx"] Jul 23 19:16:33.569453 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.569429 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2"] Jul 23 19:16:33.580947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.580925 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7"] Jul 23 19:16:33.581083 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.581063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.583857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.583842 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 19:16:33.584185 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.584171 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.584473 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.584452 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.584598 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.584526 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pzvpm\"" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.585384 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.587037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8b21220-6461-4e91-9760-27f3914bf6d4-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.587163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwsw\" (UniqueName: \"kubernetes.io/projected/a909b563-360e-4e57-9755-36fcd9a261cc-kube-api-access-4xwsw\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.587871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-config\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.587928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ncc\" (UniqueName: \"kubernetes.io/projected/5e3c871f-ef43-4047-9846-06219ca2d2d4-kube-api-access-v8ncc\") pod \"network-check-source-7bc5b6bf85-mb2nf\" (UID: \"5e3c871f-ef43-4047-9846-06219ca2d2d4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.587979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttxh\" (UniqueName: \"kubernetes.io/projected/10489313-35b1-47b5-a9b0-1e2df4a66bf6-kube-api-access-9ttxh\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6n2\" (UniqueName: \"kubernetes.io/projected/a8b21220-6461-4e91-9760-27f3914bf6d4-kube-api-access-pz6n2\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldcm\" (UniqueName: \"kubernetes.io/projected/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-kube-api-access-vldcm\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a909b563-360e-4e57-9755-36fcd9a261cc-config\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e194a48c-e7dc-495e-a269-dc0a77b33f32-snapshots\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a909b563-360e-4e57-9755-36fcd9a261cc-serving-cert\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e194a48c-e7dc-495e-a269-dc0a77b33f32-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9bn\" (UniqueName: \"kubernetes.io/projected/e194a48c-e7dc-495e-a269-dc0a77b33f32-kube-api-access-xk9bn\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.589044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6955e1ef-192b-4f01-9d94-ec04d7e747b6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.588464 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:34.088434777 +0000 UTC m=+41.661055315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : configmap references non-existent config key: service-ca.crt Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-serving-cert\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8b21220-6461-4e91-9760-27f3914bf6d4-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e194a48c-e7dc-495e-a269-dc0a77b33f32-serving-cert\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e194a48c-e7dc-495e-a269-dc0a77b33f32-service-ca-bundle\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a909b563-360e-4e57-9755-36fcd9a261cc-trusted-ca\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-stats-auth\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e194a48c-e7dc-495e-a269-dc0a77b33f32-tmp\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-default-certificate\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.588936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcxm\" (UniqueName: \"kubernetes.io/projected/6955e1ef-192b-4f01-9d94-ec04d7e747b6-kube-api-access-vqcxm\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.589163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8b21220-6461-4e91-9760-27f3914bf6d4-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.590114 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.590001 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:16:33.590836 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.590107 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:34.090087784 +0000 UTC m=+41.662708320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : secret "router-metrics-certs-default" not found Jul 23 19:16:33.591794 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.591758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8b21220-6461-4e91-9760-27f3914bf6d4-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.591977 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.591961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-stats-auth\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.592420 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.592398 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-default-certificate\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.599073 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.599050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:33.601518 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.601486 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s7pzs\"" Jul 23 19:16:33.601664 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.601525 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 19:16:33.601892 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.601876 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 19:16:33.602443 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.602425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2"] Jul 23 19:16:33.603270 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.603252 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:16:33.603334 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.603318 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7"] Jul 23 19:16:33.608440 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.608419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6n2\" (UniqueName: \"kubernetes.io/projected/a8b21220-6461-4e91-9760-27f3914bf6d4-kube-api-access-pz6n2\") pod \"kube-storage-version-migrator-operator-7658d9b749-nmxnw\" (UID: \"a8b21220-6461-4e91-9760-27f3914bf6d4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.611220 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.611189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttxh\" (UniqueName: \"kubernetes.io/projected/10489313-35b1-47b5-a9b0-1e2df4a66bf6-kube-api-access-9ttxh\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:33.689218 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e194a48c-e7dc-495e-a269-dc0a77b33f32-serving-cert\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e194a48c-e7dc-495e-a269-dc0a77b33f32-service-ca-bundle\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a909b563-360e-4e57-9755-36fcd9a261cc-trusted-ca\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e194a48c-e7dc-495e-a269-dc0a77b33f32-tmp\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcxm\" (UniqueName: \"kubernetes.io/projected/6955e1ef-192b-4f01-9d94-ec04d7e747b6-kube-api-access-vqcxm\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwsw\" (UniqueName: \"kubernetes.io/projected/a909b563-360e-4e57-9755-36fcd9a261cc-kube-api-access-4xwsw\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.689387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-config\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.689723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ncc\" (UniqueName: \"kubernetes.io/projected/5e3c871f-ef43-4047-9846-06219ca2d2d4-kube-api-access-v8ncc\") pod \"network-check-source-7bc5b6bf85-mb2nf\" (UID: \"5e3c871f-ef43-4047-9846-06219ca2d2d4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" Jul 23 19:16:33.689806 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldcm\" (UniqueName: \"kubernetes.io/projected/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-kube-api-access-vldcm\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.689859 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a909b563-360e-4e57-9755-36fcd9a261cc-config\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.689913 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e194a48c-e7dc-495e-a269-dc0a77b33f32-snapshots\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.689913 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e194a48c-e7dc-495e-a269-dc0a77b33f32-tmp\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.690011 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a909b563-360e-4e57-9755-36fcd9a261cc-serving-cert\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.690011 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e194a48c-e7dc-495e-a269-dc0a77b33f32-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.690011 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.689991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9bn\" (UniqueName: \"kubernetes.io/projected/e194a48c-e7dc-495e-a269-dc0a77b33f32-kube-api-access-xk9bn\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.690293 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e194a48c-e7dc-495e-a269-dc0a77b33f32-service-ca-bundle\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.690293 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6955e1ef-192b-4f01-9d94-ec04d7e747b6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.690293 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.690293 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmg8p\" (UniqueName: \"kubernetes.io/projected/470c3e2a-e03c-4163-95ca-c268e9544130-kube-api-access-vmg8p\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:33.690293 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-serving-cert\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.690569 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a909b563-360e-4e57-9755-36fcd9a261cc-trusted-ca\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.690569 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690558 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e194a48c-e7dc-495e-a269-dc0a77b33f32-snapshots\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.690668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-config\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.690668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690613 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a909b563-360e-4e57-9755-36fcd9a261cc-config\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.690668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" Jul 23 19:16:33.690834 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.690665 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:33.690834 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.690753 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls podName:6955e1ef-192b-4f01-9d94-ec04d7e747b6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:34.190715949 +0000 UTC m=+41.763336474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-kc7v2" (UID: "6955e1ef-192b-4f01-9d94-ec04d7e747b6") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:33.690958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6955e1ef-192b-4f01-9d94-ec04d7e747b6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.690958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.690867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e194a48c-e7dc-495e-a269-dc0a77b33f32-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.691941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.691920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e194a48c-e7dc-495e-a269-dc0a77b33f32-serving-cert\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.692415 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.692398 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-serving-cert\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.692600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.692583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a909b563-360e-4e57-9755-36fcd9a261cc-serving-cert\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.703084 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.703062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwsw\" (UniqueName: \"kubernetes.io/projected/a909b563-360e-4e57-9755-36fcd9a261cc-kube-api-access-4xwsw\") pod \"console-operator-565b7b569d-5zzxw\" (UID: \"a909b563-360e-4e57-9755-36fcd9a261cc\") " pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.704109 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.704083 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ncc\" (UniqueName: \"kubernetes.io/projected/5e3c871f-ef43-4047-9846-06219ca2d2d4-kube-api-access-v8ncc\") pod \"network-check-source-7bc5b6bf85-mb2nf\" (UID: \"5e3c871f-ef43-4047-9846-06219ca2d2d4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" Jul 23 19:16:33.704472 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.704454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcxm\" (UniqueName: \"kubernetes.io/projected/6955e1ef-192b-4f01-9d94-ec04d7e747b6-kube-api-access-vqcxm\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:33.704978 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.704961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldcm\" (UniqueName: \"kubernetes.io/projected/4300a1a8-fedf-4191-8cd6-e5818ebbe3cf-kube-api-access-vldcm\") pod \"service-ca-operator-5bcbc94c78-n6trx\" (UID: \"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.705262 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.705242 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9bn\" (UniqueName: \"kubernetes.io/projected/e194a48c-e7dc-495e-a269-dc0a77b33f32-kube-api-access-xk9bn\") pod \"insights-operator-555d9d7885-cr79v\" (UID: \"e194a48c-e7dc-495e-a269-dc0a77b33f32\") " pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.715035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.715022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" Jul 23 19:16:33.791142 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.790780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:33.791142 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.790941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmg8p\" (UniqueName: \"kubernetes.io/projected/470c3e2a-e03c-4163-95ca-c268e9544130-kube-api-access-vmg8p\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:33.791564 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.791404 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:16:33.791564 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:33.791466 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls podName:470c3e2a-e03c-4163-95ca-c268e9544130 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:34.291447882 +0000 UTC m=+41.864068420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-j7dt7" (UID: "470c3e2a-e03c-4163-95ca-c268e9544130") : secret "samples-operator-tls" not found Jul 23 19:16:33.801494 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.801267 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" Jul 23 19:16:33.824184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.823387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:33.824184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.824117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmg8p\" (UniqueName: \"kubernetes.io/projected/470c3e2a-e03c-4163-95ca-c268e9544130-kube-api-access-vmg8p\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:33.829530 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.829478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-cr79v" Jul 23 19:16:33.862394 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.862342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" Jul 23 19:16:33.866856 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.866744 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq"] Jul 23 19:16:33.870218 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:33.870192 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e34a4ee_f7cc_4171_ad03_75227ccbc291.slice/crio-ededbb188446422a644e8cd563b7eb1d7e3b2d9cce7adf9e579e286c9660c9f1 WatchSource:0}: Error finding container ededbb188446422a644e8cd563b7eb1d7e3b2d9cce7adf9e579e286c9660c9f1: Status 404 returned error can't find the container with id ededbb188446422a644e8cd563b7eb1d7e3b2d9cce7adf9e579e286c9660c9f1 Jul 23 19:16:33.906157 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.906103 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw"] Jul 23 19:16:33.911813 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:33.911785 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b21220_6461_4e91_9760_27f3914bf6d4.slice/crio-a6a20d138bea448a87a23be543bbcfd863c7e4814df8d6fdce4e846d7260d23b WatchSource:0}: Error finding container a6a20d138bea448a87a23be543bbcfd863c7e4814df8d6fdce4e846d7260d23b: Status 404 returned error can't find the container with id a6a20d138bea448a87a23be543bbcfd863c7e4814df8d6fdce4e846d7260d23b Jul 23 19:16:33.983794 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:33.983694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf"] Jul 23 19:16:33.987786 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:33.987696 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3c871f_ef43_4047_9846_06219ca2d2d4.slice/crio-0d2a410a2a0c2618b955cc6f946ed74a6a812dcad251ae733eaf0a5df02f73e8 WatchSource:0}: Error finding container 0d2a410a2a0c2618b955cc6f946ed74a6a812dcad251ae733eaf0a5df02f73e8: Status 404 returned error can't find the container with id 0d2a410a2a0c2618b955cc6f946ed74a6a812dcad251ae733eaf0a5df02f73e8 Jul 23 19:16:34.018687 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.018626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5zzxw"] Jul 23 19:16:34.023607 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:34.023542 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda909b563_360e_4e57_9755_36fcd9a261cc.slice/crio-6a4c1a36ce793804a401142cf5d926d6e02ffa38ed1f8c0b5c9e5ab0348e8e2d WatchSource:0}: Error finding container 6a4c1a36ce793804a401142cf5d926d6e02ffa38ed1f8c0b5c9e5ab0348e8e2d: Status 404 returned error can't find the container with id 6a4c1a36ce793804a401142cf5d926d6e02ffa38ed1f8c0b5c9e5ab0348e8e2d Jul 23 19:16:34.070948 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.070917 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx"] Jul 23 19:16:34.074147 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:34.074104 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4300a1a8_fedf_4191_8cd6_e5818ebbe3cf.slice/crio-61df504f2114b134c564b7572b95317a31f5a628f10a3a07e230aeac5dace983 WatchSource:0}: Error finding container 61df504f2114b134c564b7572b95317a31f5a628f10a3a07e230aeac5dace983: Status 404 returned error can't find the container with id 61df504f2114b134c564b7572b95317a31f5a628f10a3a07e230aeac5dace983 Jul 23 19:16:34.095208 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.095127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:34.095367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.095304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:34.095367 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.095352 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:35.095321144 +0000 UTC m=+42.667941666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : configmap references non-existent config key: service-ca.crt Jul 23 19:16:34.095903 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.095408 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:16:34.095903 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.095458 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:35.095442062 +0000 UTC m=+42.668062600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : secret "router-metrics-certs-default" not found Jul 23 19:16:34.096558 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.096525 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cr79v"] Jul 23 19:16:34.099076 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:34.099052 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode194a48c_e7dc_495e_a269_dc0a77b33f32.slice/crio-a29a60dec5e86a9d2bd292dacbba8a51286159035ac45a0c29fa258cb8d96415 WatchSource:0}: Error finding container a29a60dec5e86a9d2bd292dacbba8a51286159035ac45a0c29fa258cb8d96415: Status 404 returned error can't find the container with id a29a60dec5e86a9d2bd292dacbba8a51286159035ac45a0c29fa258cb8d96415 Jul 23 19:16:34.143534 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.143490 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" event={"ID":"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf","Type":"ContainerStarted","Data":"61df504f2114b134c564b7572b95317a31f5a628f10a3a07e230aeac5dace983"} Jul 23 19:16:34.144687 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.144661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" event={"ID":"a909b563-360e-4e57-9755-36fcd9a261cc","Type":"ContainerStarted","Data":"6a4c1a36ce793804a401142cf5d926d6e02ffa38ed1f8c0b5c9e5ab0348e8e2d"} Jul 23 19:16:34.145667 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.145646 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" event={"ID":"5e3c871f-ef43-4047-9846-06219ca2d2d4","Type":"ContainerStarted","Data":"0d2a410a2a0c2618b955cc6f946ed74a6a812dcad251ae733eaf0a5df02f73e8"} Jul 23 19:16:34.146798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.146773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" event={"ID":"6e34a4ee-f7cc-4171-ad03-75227ccbc291","Type":"ContainerStarted","Data":"ededbb188446422a644e8cd563b7eb1d7e3b2d9cce7adf9e579e286c9660c9f1"} Jul 23 19:16:34.147791 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.147772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cr79v" event={"ID":"e194a48c-e7dc-495e-a269-dc0a77b33f32","Type":"ContainerStarted","Data":"a29a60dec5e86a9d2bd292dacbba8a51286159035ac45a0c29fa258cb8d96415"} Jul 23 19:16:34.148869 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.148849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" event={"ID":"a8b21220-6461-4e91-9760-27f3914bf6d4","Type":"ContainerStarted","Data":"a6a20d138bea448a87a23be543bbcfd863c7e4814df8d6fdce4e846d7260d23b"} Jul 23 19:16:34.196915 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.196881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:34.197068 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.196997 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:34.197068 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.197050 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls podName:6955e1ef-192b-4f01-9d94-ec04d7e747b6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:35.197036697 +0000 UTC m=+42.769657218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-kc7v2" (UID: "6955e1ef-192b-4f01-9d94-ec04d7e747b6") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:34.298134 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:34.298055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:34.298268 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.298201 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:16:34.298268 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:34.298267 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls podName:470c3e2a-e03c-4163-95ca-c268e9544130 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:35.298252293 +0000 UTC m=+42.870872812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-j7dt7" (UID: "470c3e2a-e03c-4163-95ca-c268e9544130") : secret "samples-operator-tls" not found Jul 23 19:16:35.106465 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:35.105586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:35.106465 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:35.105700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:35.106465 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.105880 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:16:35.106465 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.105944 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:37.105925545 +0000 UTC m=+44.678546066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : secret "router-metrics-certs-default" not found Jul 23 19:16:35.106465 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.106363 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:37.10634462 +0000 UTC m=+44.678965143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : configmap references non-existent config key: service-ca.crt Jul 23 19:16:35.207093 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:35.207051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:35.207547 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.207262 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:35.207547 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.207330 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls podName:6955e1ef-192b-4f01-9d94-ec04d7e747b6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:37.207310758 +0000 UTC m=+44.779931290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-kc7v2" (UID: "6955e1ef-192b-4f01-9d94-ec04d7e747b6") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:35.308151 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:35.308108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:35.308450 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.308429 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:16:35.308528 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:35.308504 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls podName:470c3e2a-e03c-4163-95ca-c268e9544130 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:37.30848347 +0000 UTC m=+44.881104003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-j7dt7" (UID: "470c3e2a-e03c-4163-95ca-c268e9544130") : secret "samples-operator-tls" not found Jul 23 19:16:36.622642 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:36.622553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:36.623096 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:36.622658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:36.623096 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:36.622793 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:16:36.623096 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:36.622821 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578c5786db-p5fc7: secret "image-registry-tls" not found Jul 23 19:16:36.623096 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:36.622895 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls podName:22e2c84f-1b54-4bed-8c78-5ebf0253c790 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:44.622873047 +0000 UTC m=+52.195493576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls") pod "image-registry-578c5786db-p5fc7" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790") : secret "image-registry-tls" not found Jul 23 19:16:36.623096 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:36.622894 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:16:36.623096 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:36.622958 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls podName:21caab7f-0bf4-44bd-8a75-21db01239fb5 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:44.622944631 +0000 UTC m=+52.195565148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls") pod "dns-default-n4v72" (UID: "21caab7f-0bf4-44bd-8a75-21db01239fb5") : secret "dns-default-metrics-tls" not found Jul 23 19:16:37.127210 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:37.127165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:37.127386 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:37.127283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:37.127386 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.127346 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:41.127324333 +0000 UTC m=+48.699944863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : configmap references non-existent config key: service-ca.crt Jul 23 19:16:37.127526 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.127414 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:16:37.127526 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.127482 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:41.127465277 +0000 UTC m=+48.700085796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : secret "router-metrics-certs-default" not found Jul 23 19:16:37.228421 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:37.228380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:37.228562 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.228528 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:37.228609 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.228598 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls podName:6955e1ef-192b-4f01-9d94-ec04d7e747b6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:41.228578335 +0000 UTC m=+48.801198856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-kc7v2" (UID: "6955e1ef-192b-4f01-9d94-ec04d7e747b6") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:37.328981 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:37.328944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:37.329135 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.329102 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:16:37.329189 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:37.329168 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls podName:470c3e2a-e03c-4163-95ca-c268e9544130 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:41.329151348 +0000 UTC m=+48.901771865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-j7dt7" (UID: "470c3e2a-e03c-4163-95ca-c268e9544130") : secret "samples-operator-tls" not found Jul 23 19:16:39.322314 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.322283 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs"] Jul 23 19:16:39.361351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.361315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs"] Jul 23 19:16:39.361534 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.361467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:39.364534 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.364508 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6r77j\"" Jul 23 19:16:39.364665 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.364508 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 19:16:39.364665 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.364508 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 19:16:39.447623 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.447586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:39.447817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.447717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1db54a43-75dd-4d51-95ac-b95144a6f0fb-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:39.548990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.548956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:39.549174 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.549099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1db54a43-75dd-4d51-95ac-b95144a6f0fb-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:39.549174 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:39.549114 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:16:39.549279 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:39.549226 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert podName:1db54a43-75dd-4d51-95ac-b95144a6f0fb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:40.049207878 +0000 UTC m=+47.621828397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q9qgs" (UID: "1db54a43-75dd-4d51-95ac-b95144a6f0fb") : secret "networking-console-plugin-cert" not found Jul 23 19:16:39.549862 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:39.549833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1db54a43-75dd-4d51-95ac-b95144a6f0fb-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:40.054063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:40.054023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:40.054231 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:40.054182 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:16:40.054276 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:40.054247 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert podName:1db54a43-75dd-4d51-95ac-b95144a6f0fb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:41.054231328 +0000 UTC m=+48.626851846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q9qgs" (UID: "1db54a43-75dd-4d51-95ac-b95144a6f0fb") : secret "networking-console-plugin-cert" not found Jul 23 19:16:41.064342 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.064245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:41.064788 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.064451 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:16:41.064788 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.064539 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert podName:1db54a43-75dd-4d51-95ac-b95144a6f0fb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:43.064517915 +0000 UTC m=+50.637138448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q9qgs" (UID: "1db54a43-75dd-4d51-95ac-b95144a6f0fb") : secret "networking-console-plugin-cert" not found Jul 23 19:16:41.165593 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.165550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:41.165817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.165666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:41.165817 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.165716 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:49.165696611 +0000 UTC m=+56.738317134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : configmap references non-existent config key: service-ca.crt Jul 23 19:16:41.165942 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.165847 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:16:41.165942 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.165918 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:49.165901747 +0000 UTC m=+56.738522279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : secret "router-metrics-certs-default" not found Jul 23 19:16:41.172173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.172129 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" event={"ID":"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf","Type":"ContainerStarted","Data":"e6ea7175ebbb6d5072c4935cae49eb75b12b479efb96049a45a7fa78ac7f58a3"} Jul 23 19:16:41.173973 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.173950 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/0.log" Jul 23 19:16:41.174077 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.173994 2574 generic.go:356] "Generic (PLEG): container finished" podID="a909b563-360e-4e57-9755-36fcd9a261cc" containerID="e4998ed74666e59f2dac79451d169650f0397f0817ebd0d6675c70762da10f7d" exitCode=255 Jul 23 19:16:41.174138 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.174080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" event={"ID":"a909b563-360e-4e57-9755-36fcd9a261cc","Type":"ContainerDied","Data":"e4998ed74666e59f2dac79451d169650f0397f0817ebd0d6675c70762da10f7d"} Jul 23 19:16:41.174282 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.174264 2574 scope.go:117] "RemoveContainer" containerID="e4998ed74666e59f2dac79451d169650f0397f0817ebd0d6675c70762da10f7d" Jul 23 19:16:41.176126 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.176026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" event={"ID":"5e3c871f-ef43-4047-9846-06219ca2d2d4","Type":"ContainerStarted","Data":"a490c4e1fb1a88600068c67a38c6d947028cb7266a4f6ec3d8bb40e34834f8da"} Jul 23 19:16:41.177684 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.177497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" event={"ID":"6e34a4ee-f7cc-4171-ad03-75227ccbc291","Type":"ContainerStarted","Data":"3812f11a68ecb58be2954621e199f8b4d3a6de73ef753e66b572c40c7b27bb29"} Jul 23 19:16:41.179274 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.179252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cr79v" event={"ID":"e194a48c-e7dc-495e-a269-dc0a77b33f32","Type":"ContainerStarted","Data":"8dd462ccf07650410f3f3a7d099628dd0fc34132a869ec34c7aaff4d4aeae7eb"} Jul 23 19:16:41.181375 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.181353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" event={"ID":"a8b21220-6461-4e91-9760-27f3914bf6d4","Type":"ContainerStarted","Data":"6e135ca7bfb0485a4b85d1b9ddd26772c58cec8835156e954051f27cf443457c"} Jul 23 19:16:41.192537 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.192480 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" podStartSLOduration=1.7414179760000001 podStartE2EDuration="8.192465639s" totalImagesPullingTime="6.451047663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:41.192077127 +0000 UTC m=+48.764697665" watchObservedRunningTime="2026-07-23 19:16:41.192465639 +0000 UTC m=+48.765086180" Jul 23 19:16:41.211432 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.211374 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hzgwq" podStartSLOduration=6.072681 podStartE2EDuration="8.211360011s" totalImagesPullingTime="2.138679011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:41.210998997 +0000 UTC m=+48.783619549" watchObservedRunningTime="2026-07-23 19:16:41.211360011 +0000 UTC m=+48.783980551" Jul 23 19:16:41.232448 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.231827 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mb2nf" podStartSLOduration=1.694451272 podStartE2EDuration="8.231807901s" totalImagesPullingTime="6.537356629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:41.230591301 +0000 UTC m=+48.803211841" watchObservedRunningTime="2026-07-23 19:16:41.231807901 +0000 UTC m=+48.804428442" Jul 23 19:16:41.266674 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.266618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:41.267134 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.267112 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:41.267327 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.267312 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls podName:6955e1ef-192b-4f01-9d94-ec04d7e747b6 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:49.267286441 +0000 UTC m=+56.839906962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-kc7v2" (UID: "6955e1ef-192b-4f01-9d94-ec04d7e747b6") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:41.310071 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.309704 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-cr79v" podStartSLOduration=1.896769793 podStartE2EDuration="8.30968261s" totalImagesPullingTime="6.412912817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:41.28786566 +0000 UTC m=+48.860486202" watchObservedRunningTime="2026-07-23 19:16:41.30968261 +0000 UTC m=+48.882303151" Jul 23 19:16:41.310615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.310568 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" podStartSLOduration=1.712308825 podStartE2EDuration="8.31055229s" totalImagesPullingTime="6.598243465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:41.309411897 +0000 UTC m=+48.882032437" watchObservedRunningTime="2026-07-23 19:16:41.31055229 +0000 UTC m=+48.883172830" Jul 23 19:16:41.367265 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:41.367227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:41.367407 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.367338 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:16:41.367407 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:41.367397 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls podName:470c3e2a-e03c-4163-95ca-c268e9544130 nodeName:}" failed. No retries permitted until 2026-07-23 19:16:49.367379129 +0000 UTC m=+56.939999665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-j7dt7" (UID: "470c3e2a-e03c-4163-95ca-c268e9544130") : secret "samples-operator-tls" not found Jul 23 19:16:42.008295 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.008254 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6n8l4"] Jul 23 19:16:42.023604 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.023573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.025696 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.025669 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6n8l4"] Jul 23 19:16:42.028790 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.028763 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 19:16:42.028922 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.028812 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 19:16:42.028922 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.028773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p4fh2\"" Jul 23 19:16:42.075217 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.075183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-crio-socket\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.075604 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.075279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.075604 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.075318 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xv4\" (UniqueName: \"kubernetes.io/projected/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-api-access-p9xv4\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.075604 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.075363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.075604 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.075478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-data-volume\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-data-volume\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176359 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-crio-socket\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176425 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176468 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xv4\" (UniqueName: \"kubernetes.io/projected/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-api-access-p9xv4\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176542 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-crio-socket\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176594 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.176538 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:16:42.176594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-data-volume\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.176569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.176755 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.176707 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:16:42.176755 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.176716 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:42.676697531 +0000 UTC m=+50.249318053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : configmap "kube-rbac-proxy" not found Jul 23 19:16:42.176872 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.176791 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:42.676774057 +0000 UTC m=+50.249394589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : secret "insights-runtime-extractor-tls" not found Jul 23 19:16:42.185377 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.185347 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xv4\" (UniqueName: \"kubernetes.io/projected/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-api-access-p9xv4\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.186532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.186487 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:16:42.187237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.187221 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/0.log" Jul 23 19:16:42.187406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.187387 2574 generic.go:356] "Generic (PLEG): container finished" podID="a909b563-360e-4e57-9755-36fcd9a261cc" containerID="c87a12d6ede26758eacc6cd30d3b1252dc29060669b354c86029ef4e1ad33e29" exitCode=255 Jul 23 19:16:42.187764 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.187704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" event={"ID":"a909b563-360e-4e57-9755-36fcd9a261cc","Type":"ContainerDied","Data":"c87a12d6ede26758eacc6cd30d3b1252dc29060669b354c86029ef4e1ad33e29"} Jul 23 19:16:42.187863 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.187828 2574 scope.go:117] "RemoveContainer" containerID="e4998ed74666e59f2dac79451d169650f0397f0817ebd0d6675c70762da10f7d" Jul 23 19:16:42.188664 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.188650 2574 scope.go:117] "RemoveContainer" containerID="c87a12d6ede26758eacc6cd30d3b1252dc29060669b354c86029ef4e1ad33e29" Jul 23 19:16:42.189111 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.189079 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5zzxw_openshift-console-operator(a909b563-360e-4e57-9755-36fcd9a261cc)\"" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" podUID="a909b563-360e-4e57-9755-36fcd9a261cc" Jul 23 19:16:42.682610 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.682567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.682846 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:42.682629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:42.682846 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.682704 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 19:16:42.682846 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.682727 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:16:42.682846 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.682811 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:43.682788511 +0000 UTC m=+51.255409047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : configmap "kube-rbac-proxy" not found Jul 23 19:16:42.682846 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:42.682831 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:43.682822902 +0000 UTC m=+51.255443427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : secret "insights-runtime-extractor-tls" not found Jul 23 19:16:43.085020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.084933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:43.085359 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:43.085091 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:16:43.085359 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:43.085163 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert podName:1db54a43-75dd-4d51-95ac-b95144a6f0fb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:47.085145295 +0000 UTC m=+54.657765836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q9qgs" (UID: "1db54a43-75dd-4d51-95ac-b95144a6f0fb") : secret "networking-console-plugin-cert" not found Jul 23 19:16:43.192702 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.192675 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:16:43.193070 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.193052 2574 scope.go:117] "RemoveContainer" containerID="c87a12d6ede26758eacc6cd30d3b1252dc29060669b354c86029ef4e1ad33e29" Jul 23 19:16:43.193245 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:43.193225 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5zzxw_openshift-console-operator(a909b563-360e-4e57-9755-36fcd9a261cc)\"" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" podUID="a909b563-360e-4e57-9755-36fcd9a261cc" Jul 23 19:16:43.408576 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.408545 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qc484_36bf83e3-c725-41e9-b5b8-2b79ec57b7a7/dns-node-resolver/0.log" Jul 23 19:16:43.691484 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.691395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:43.691484 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.691455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:43.691723 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:43.691705 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:16:43.691852 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:43.691789 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:45.691775193 +0000 UTC m=+53.264395711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : secret "insights-runtime-extractor-tls" not found Jul 23 19:16:43.691996 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.691977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:43.823715 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.823676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:43.823879 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:43.823750 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:44.002999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:44.002927 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xn7q_8d4319b6-feb5-4089-8896-3e93d2e2e099/node-ca/0.log" Jul 23 19:16:44.195688 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:44.195662 2574 scope.go:117] "RemoveContainer" containerID="c87a12d6ede26758eacc6cd30d3b1252dc29060669b354c86029ef4e1ad33e29" Jul 23 19:16:44.196078 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:44.195825 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5zzxw_openshift-console-operator(a909b563-360e-4e57-9755-36fcd9a261cc)\"" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" podUID="a909b563-360e-4e57-9755-36fcd9a261cc" Jul 23 19:16:44.700603 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:44.700565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:16:44.700799 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:44.700693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:16:44.700799 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:44.700724 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 19:16:44.700880 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:44.700810 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls podName:21caab7f-0bf4-44bd-8a75-21db01239fb5 nodeName:}" failed. No retries permitted until 2026-07-23 19:17:00.700791582 +0000 UTC m=+68.273412100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls") pod "dns-default-n4v72" (UID: "21caab7f-0bf4-44bd-8a75-21db01239fb5") : secret "dns-default-metrics-tls" not found Jul 23 19:16:44.700880 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:44.700826 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 19:16:44.700880 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:44.700837 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-578c5786db-p5fc7: secret "image-registry-tls" not found Jul 23 19:16:44.700976 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:44.700882 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls podName:22e2c84f-1b54-4bed-8c78-5ebf0253c790 nodeName:}" failed. No retries permitted until 2026-07-23 19:17:00.700870659 +0000 UTC m=+68.273491179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls") pod "image-registry-578c5786db-p5fc7" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790") : secret "image-registry-tls" not found Jul 23 19:16:44.809190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:44.809162 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hj9rw_24f25159-b9cc-4653-8ed2-d54b0a01f3e0/serve-healthcheck-canary/0.log" Jul 23 19:16:45.711087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:45.711046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:45.711539 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:45.711171 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:16:45.711539 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:45.711225 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:49.711211649 +0000 UTC m=+57.283832166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : secret "insights-runtime-extractor-tls" not found Jul 23 19:16:47.124689 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:47.124647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:47.125076 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:47.124830 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:16:47.125076 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:47.124915 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert podName:1db54a43-75dd-4d51-95ac-b95144a6f0fb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:55.124896456 +0000 UTC m=+62.697516973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q9qgs" (UID: "1db54a43-75dd-4d51-95ac-b95144a6f0fb") : secret "networking-console-plugin-cert" not found Jul 23 19:16:49.243237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:49.243197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:49.243671 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:49.243304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:16:49.243671 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.243377 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 19:16:49.243671 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.243430 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:17:05.243416628 +0000 UTC m=+72.816037146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : configmap references non-existent config key: service-ca.crt Jul 23 19:16:49.243671 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.243445 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs podName:10489313-35b1-47b5-a9b0-1e2df4a66bf6 nodeName:}" failed. No retries permitted until 2026-07-23 19:17:05.243438594 +0000 UTC m=+72.816059111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs") pod "router-default-64cbdc95dd-q7nph" (UID: "10489313-35b1-47b5-a9b0-1e2df4a66bf6") : secret "router-metrics-certs-default" not found Jul 23 19:16:49.344054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:49.344016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:16:49.344247 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.344186 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:49.344310 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.344266 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls podName:6955e1ef-192b-4f01-9d94-ec04d7e747b6 nodeName:}" failed. No retries permitted until 2026-07-23 19:17:05.34424648 +0000 UTC m=+72.916866999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-kc7v2" (UID: "6955e1ef-192b-4f01-9d94-ec04d7e747b6") : secret "cluster-monitoring-operator-tls" not found Jul 23 19:16:49.444684 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:49.444645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:16:49.444881 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.444836 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 19:16:49.444918 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.444912 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls podName:470c3e2a-e03c-4163-95ca-c268e9544130 nodeName:}" failed. No retries permitted until 2026-07-23 19:17:05.444895085 +0000 UTC m=+73.017515606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-j7dt7" (UID: "470c3e2a-e03c-4163-95ca-c268e9544130") : secret "samples-operator-tls" not found Jul 23 19:16:49.746916 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:49.746876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:49.747129 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.747063 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 19:16:49.747194 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:49.747162 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls podName:f3869c9d-31a5-437f-96ae-a1f7c54f07cb nodeName:}" failed. No retries permitted until 2026-07-23 19:16:57.747140854 +0000 UTC m=+65.319761391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6n8l4" (UID: "f3869c9d-31a5-437f-96ae-a1f7c54f07cb") : secret "insights-runtime-extractor-tls" not found Jul 23 19:16:54.112458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:54.112428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49l7x" Jul 23 19:16:55.198329 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:55.198285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:16:55.198797 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:55.198448 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 19:16:55.198797 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:16:55.198530 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert podName:1db54a43-75dd-4d51-95ac-b95144a6f0fb nodeName:}" failed. No retries permitted until 2026-07-23 19:17:11.198514133 +0000 UTC m=+78.771134655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-q9qgs" (UID: "1db54a43-75dd-4d51-95ac-b95144a6f0fb") : secret "networking-console-plugin-cert" not found Jul 23 19:16:57.823378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:57.823342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:57.825871 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:57.825839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3869c9d-31a5-437f-96ae-a1f7c54f07cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6n8l4\" (UID: \"f3869c9d-31a5-437f-96ae-a1f7c54f07cb\") " pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:57.841038 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:57.841016 2574 scope.go:117] "RemoveContainer" containerID="c87a12d6ede26758eacc6cd30d3b1252dc29060669b354c86029ef4e1ad33e29" Jul 23 19:16:57.938701 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:57.938667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p4fh2\"" Jul 23 19:16:57.946049 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:57.946020 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6n8l4" Jul 23 19:16:58.079379 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.079305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6n8l4"] Jul 23 19:16:58.082147 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:58.082117 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3869c9d_31a5_437f_96ae_a1f7c54f07cb.slice/crio-2cd8b1a5fa3e94c1e551404361681b224a54ffed4f6ec1d04d18df300018821f WatchSource:0}: Error finding container 2cd8b1a5fa3e94c1e551404361681b224a54ffed4f6ec1d04d18df300018821f: Status 404 returned error can't find the container with id 2cd8b1a5fa3e94c1e551404361681b224a54ffed4f6ec1d04d18df300018821f Jul 23 19:16:58.239980 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.239947 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:16:58.240162 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.240060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" event={"ID":"a909b563-360e-4e57-9755-36fcd9a261cc","Type":"ContainerStarted","Data":"c53345018f1c970ecb45c536a6c990d4aff27360e9e1459991255ff974fc0a41"} Jul 23 19:16:58.240510 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.240482 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:58.241420 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.241394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n8l4" event={"ID":"f3869c9d-31a5-437f-96ae-a1f7c54f07cb","Type":"ContainerStarted","Data":"2cd8b1a5fa3e94c1e551404361681b224a54ffed4f6ec1d04d18df300018821f"} Jul 23 19:16:58.246936 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.246899 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" Jul 23 19:16:58.262799 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.262725 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-5zzxw" podStartSLOduration=18.781278826 podStartE2EDuration="25.262710951s" totalImagesPullingTime="6.481432125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:58.26243362 +0000 UTC m=+65.835054161" watchObservedRunningTime="2026-07-23 19:16:58.262710951 +0000 UTC m=+65.835331491" Jul 23 19:16:58.632210 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.632167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:58.641025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.637376 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 19:16:58.646283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.646225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b0171d6-8abc-4a3e-a72a-f29271153a70-original-pull-secret\") pod \"global-pull-secret-syncer-2gqgt\" (UID: \"6b0171d6-8abc-4a3e-a72a-f29271153a70\") " pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:58.733809 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.733706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:58.734001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.733898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:58.737612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.737581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78fd\" (UniqueName: \"kubernetes.io/projected/5eefc014-bc40-4c10-8246-6f7ae61cfa1b-kube-api-access-x78fd\") pod \"network-check-target-x5szb\" (UID: \"5eefc014-bc40-4c10-8246-6f7ae61cfa1b\") " pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:58.738342 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.738126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 19:16:58.750723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.750654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ab9bc48-2193-4381-848e-fb7f5ba5fca0-metrics-certs\") pod \"network-metrics-daemon-8679f\" (UID: \"3ab9bc48-2193-4381-848e-fb7f5ba5fca0\") " pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:58.858956 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.858745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6wbwl\"" Jul 23 19:16:58.863763 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.863578 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2gqgt" Jul 23 19:16:58.866579 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.866558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8679f" Jul 23 19:16:58.871876 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.871852 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v5npk\"" Jul 23 19:16:58.880759 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:58.880513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:59.071082 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.070992 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5szb"] Jul 23 19:16:59.075104 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:59.075073 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eefc014_bc40_4c10_8246_6f7ae61cfa1b.slice/crio-863853c7aa324a111fe391d26df979648742baf095e9b4870d1685dc8cc89847 WatchSource:0}: Error finding container 863853c7aa324a111fe391d26df979648742baf095e9b4870d1685dc8cc89847: Status 404 returned error can't find the container with id 863853c7aa324a111fe391d26df979648742baf095e9b4870d1685dc8cc89847 Jul 23 19:16:59.092005 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.091981 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2gqgt"] Jul 23 19:16:59.093612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.093542 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8679f"] Jul 23 19:16:59.094626 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:59.094565 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab9bc48_2193_4381_848e_fb7f5ba5fca0.slice/crio-e457cf2b9643bb4abbc3d21efcb322803c3a857e4ff308acf223e7c036bae3fb WatchSource:0}: Error finding container e457cf2b9643bb4abbc3d21efcb322803c3a857e4ff308acf223e7c036bae3fb: Status 404 returned error can't find the container with id e457cf2b9643bb4abbc3d21efcb322803c3a857e4ff308acf223e7c036bae3fb Jul 23 19:16:59.099055 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:16:59.098096 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0171d6_8abc_4a3e_a72a_f29271153a70.slice/crio-4f128c98eec4e6b98a00dbb2e85bf21723a3ccb7adc01f363e6c3fcc45bc945d WatchSource:0}: Error finding container 4f128c98eec4e6b98a00dbb2e85bf21723a3ccb7adc01f363e6c3fcc45bc945d: Status 404 returned error can't find the container with id 4f128c98eec4e6b98a00dbb2e85bf21723a3ccb7adc01f363e6c3fcc45bc945d Jul 23 19:16:59.246810 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.246772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5szb" event={"ID":"5eefc014-bc40-4c10-8246-6f7ae61cfa1b","Type":"ContainerStarted","Data":"04d8447235c3c409d8c13fb875709a77e5516ad755c3a1b160ade0a49ea0730a"} Jul 23 19:16:59.247000 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.246835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5szb" event={"ID":"5eefc014-bc40-4c10-8246-6f7ae61cfa1b","Type":"ContainerStarted","Data":"863853c7aa324a111fe391d26df979648742baf095e9b4870d1685dc8cc89847"} Jul 23 19:16:59.247000 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.246863 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:16:59.248035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.248010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8679f" event={"ID":"3ab9bc48-2193-4381-848e-fb7f5ba5fca0","Type":"ContainerStarted","Data":"e457cf2b9643bb4abbc3d21efcb322803c3a857e4ff308acf223e7c036bae3fb"} Jul 23 19:16:59.249176 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.249154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2gqgt" event={"ID":"6b0171d6-8abc-4a3e-a72a-f29271153a70","Type":"ContainerStarted","Data":"4f128c98eec4e6b98a00dbb2e85bf21723a3ccb7adc01f363e6c3fcc45bc945d"} Jul 23 19:16:59.264256 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:16:59.264207 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x5szb" podStartSLOduration=66.264189027 podStartE2EDuration="1m6.264189027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:16:59.262843326 +0000 UTC m=+66.835463867" watchObservedRunningTime="2026-07-23 19:16:59.264189027 +0000 UTC m=+66.836809569" Jul 23 19:17:00.757830 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.757794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:00.758402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.757891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:17:00.760717 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.760683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21caab7f-0bf4-44bd-8a75-21db01239fb5-metrics-tls\") pod \"dns-default-n4v72\" (UID: \"21caab7f-0bf4-44bd-8a75-21db01239fb5\") " pod="openshift-dns/dns-default-n4v72" Jul 23 19:17:00.761022 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.760998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"image-registry-578c5786db-p5fc7\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:00.906991 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.906949 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p6k4c\"" Jul 23 19:17:00.914260 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.914232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:00.918226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.918197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gvmps\"" Jul 23 19:17:00.926167 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:00.926143 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n4v72" Jul 23 19:17:01.259365 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:01.259330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8679f" event={"ID":"3ab9bc48-2193-4381-848e-fb7f5ba5fca0","Type":"ContainerStarted","Data":"f3cecac85ad27c4dd3056524d6916e59e0176b806c790424a2295c852c166980"} Jul 23 19:17:01.259543 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:01.259396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8679f" event={"ID":"3ab9bc48-2193-4381-848e-fb7f5ba5fca0","Type":"ContainerStarted","Data":"07fbebe1ca872d567671d5a7e4bc1ad19e7cdd37886e883568067ca430b00b8b"} Jul 23 19:17:01.303785 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:01.303702 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8679f" podStartSLOduration=67.234481526 podStartE2EDuration="1m8.303676806s" totalImagesPullingTime="1.06919528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:01.301696687 +0000 UTC m=+68.874317227" watchObservedRunningTime="2026-07-23 19:17:01.303676806 +0000 UTC m=+68.876297348" Jul 23 19:17:05.299193 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.299151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:05.299689 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.299229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:05.299913 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.299892 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10489313-35b1-47b5-a9b0-1e2df4a66bf6-service-ca-bundle\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:05.301999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.301966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10489313-35b1-47b5-a9b0-1e2df4a66bf6-metrics-certs\") pod \"router-default-64cbdc95dd-q7nph\" (UID: \"10489313-35b1-47b5-a9b0-1e2df4a66bf6\") " pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:05.400402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.400360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:17:05.402906 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.402877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6955e1ef-192b-4f01-9d94-ec04d7e747b6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-kc7v2\" (UID: \"6955e1ef-192b-4f01-9d94-ec04d7e747b6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:17:05.501096 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.501055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:17:05.504053 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.504022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/470c3e2a-e03c-4163-95ca-c268e9544130-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-j7dt7\" (UID: \"470c3e2a-e03c-4163-95ca-c268e9544130\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:17:05.544281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.544248 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-b6gw9\"" Jul 23 19:17:05.551628 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.551561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:05.699554 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.698917 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pzvpm\"" Jul 23 19:17:05.704854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.704612 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" Jul 23 19:17:05.718216 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.718156 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s7pzs\"" Jul 23 19:17:05.720288 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.720222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" Jul 23 19:17:05.775622 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.775549 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64cbdc95dd-q7nph"] Jul 23 19:17:05.783220 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:17:05.783145 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10489313_35b1_47b5_a9b0_1e2df4a66bf6.slice/crio-2f148eaa24c88208220d284d32ac1fafc2457b1b81f199a07775c20b20c8668b WatchSource:0}: Error finding container 2f148eaa24c88208220d284d32ac1fafc2457b1b81f199a07775c20b20c8668b: Status 404 returned error can't find the container with id 2f148eaa24c88208220d284d32ac1fafc2457b1b81f199a07775c20b20c8668b Jul 23 19:17:05.805498 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.805376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n4v72"] Jul 23 19:17:05.812137 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:17:05.812098 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21caab7f_0bf4_44bd_8a75_21db01239fb5.slice/crio-01f1648bda5d6319fd482dff9e272147a40f6455ceb0e258a4de004c3934a085 WatchSource:0}: Error finding container 01f1648bda5d6319fd482dff9e272147a40f6455ceb0e258a4de004c3934a085: Status 404 returned error can't find the container with id 01f1648bda5d6319fd482dff9e272147a40f6455ceb0e258a4de004c3934a085 Jul 23 19:17:05.843564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.843536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-578c5786db-p5fc7"] Jul 23 19:17:05.848918 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:17:05.848878 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e2c84f_1b54_4bed_8c78_5ebf0253c790.slice/crio-67f0a5093d39d107908cb6b4e8b1bfce734c5c028c575a6944d60c5e65b4127c WatchSource:0}: Error finding container 67f0a5093d39d107908cb6b4e8b1bfce734c5c028c575a6944d60c5e65b4127c: Status 404 returned error can't find the container with id 67f0a5093d39d107908cb6b4e8b1bfce734c5c028c575a6944d60c5e65b4127c Jul 23 19:17:05.901337 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.901245 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2"] Jul 23 19:17:05.906135 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:17:05.906103 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6955e1ef_192b_4f01_9d94_ec04d7e747b6.slice/crio-39c5e466efd34c610fcd64d675fcff6e95ee6aaf3b6606ac91cb0897bfc4703b WatchSource:0}: Error finding container 39c5e466efd34c610fcd64d675fcff6e95ee6aaf3b6606ac91cb0897bfc4703b: Status 404 returned error can't find the container with id 39c5e466efd34c610fcd64d675fcff6e95ee6aaf3b6606ac91cb0897bfc4703b Jul 23 19:17:05.926946 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:05.926886 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7"] Jul 23 19:17:06.281998 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.281956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2gqgt" event={"ID":"6b0171d6-8abc-4a3e-a72a-f29271153a70","Type":"ContainerStarted","Data":"f134493a03c1889ec404f5d283893b72f620e2e09a2dbe00f850bd79a24c59c9"} Jul 23 19:17:06.283293 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.283261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" event={"ID":"470c3e2a-e03c-4163-95ca-c268e9544130","Type":"ContainerStarted","Data":"0e9e5b3a96422032cf0ab722b184afedb4694feacf155a51e41890e79540b32f"} Jul 23 19:17:06.285134 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.285058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" event={"ID":"22e2c84f-1b54-4bed-8c78-5ebf0253c790","Type":"ContainerStarted","Data":"6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5"} Jul 23 19:17:06.285134 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.285102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" event={"ID":"22e2c84f-1b54-4bed-8c78-5ebf0253c790","Type":"ContainerStarted","Data":"67f0a5093d39d107908cb6b4e8b1bfce734c5c028c575a6944d60c5e65b4127c"} Jul 23 19:17:06.285134 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.285128 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:06.286685 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.286651 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" event={"ID":"6955e1ef-192b-4f01-9d94-ec04d7e747b6","Type":"ContainerStarted","Data":"39c5e466efd34c610fcd64d675fcff6e95ee6aaf3b6606ac91cb0897bfc4703b"} Jul 23 19:17:06.287938 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.287902 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n4v72" event={"ID":"21caab7f-0bf4-44bd-8a75-21db01239fb5","Type":"ContainerStarted","Data":"01f1648bda5d6319fd482dff9e272147a40f6455ceb0e258a4de004c3934a085"} Jul 23 19:17:06.289618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.289591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n8l4" event={"ID":"f3869c9d-31a5-437f-96ae-a1f7c54f07cb","Type":"ContainerStarted","Data":"dba94791e1efb7f92afb43396a98a7d61db75272d6141b1ce22329c231a8c156"} Jul 23 19:17:06.291264 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.291233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" event={"ID":"10489313-35b1-47b5-a9b0-1e2df4a66bf6","Type":"ContainerStarted","Data":"5c4058a4e1dff1d9f3b6ce82334e000a785bde398b68cfc6c51a52aa5d317907"} Jul 23 19:17:06.291375 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.291276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" event={"ID":"10489313-35b1-47b5-a9b0-1e2df4a66bf6","Type":"ContainerStarted","Data":"2f148eaa24c88208220d284d32ac1fafc2457b1b81f199a07775c20b20c8668b"} Jul 23 19:17:06.306383 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.306322 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2gqgt" podStartSLOduration=66.797386964 podStartE2EDuration="1m13.306302296s" totalImagesPullingTime="6.508915332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:06.305958464 +0000 UTC m=+73.878579006" watchObservedRunningTime="2026-07-23 19:17:06.306302296 +0000 UTC m=+73.878923012" Jul 23 19:17:06.329294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.329230 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" podStartSLOduration=33.329212754 podStartE2EDuration="33.329212754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:06.327873954 +0000 UTC m=+73.900494494" watchObservedRunningTime="2026-07-23 19:17:06.329212754 +0000 UTC m=+73.901833306" Jul 23 19:17:06.353400 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.353062 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" podStartSLOduration=73.353034617 podStartE2EDuration="1m13.353034617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:06.351909184 +0000 UTC m=+73.924529731" watchObservedRunningTime="2026-07-23 19:17:06.353034617 +0000 UTC m=+73.925655162" Jul 23 19:17:06.552879 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.552794 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:06.556216 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:06.555989 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:07.300966 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:07.299939 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:07.301761 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:07.301539 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64cbdc95dd-q7nph" Jul 23 19:17:08.135683 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.135644 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b"] Jul 23 19:17:08.140699 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.140672 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.144775 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.144696 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 19:17:08.144917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.144781 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 19:17:08.144917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.144696 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 23 19:17:08.144917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.144702 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 23 19:17:08.145222 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.145197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 23 19:17:08.145580 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.145443 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 19:17:08.146184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.145973 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 23 19:17:08.205555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.205515 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b"] Jul 23 19:17:08.227433 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.227396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.227648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.227449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.227648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.227483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-ca\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.227648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.227561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l25s\" (UniqueName: \"kubernetes.io/projected/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-kube-api-access-6l25s\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.227648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.227639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.227870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.227684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-hub\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.328292 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.328251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.328292 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.328290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.328536 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.328338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-ca\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.328536 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.328380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l25s\" (UniqueName: \"kubernetes.io/projected/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-kube-api-access-6l25s\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.328536 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.328445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.328536 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.328497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-hub\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.330233 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.329428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.331485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.331439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.331656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.331631 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.332460 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.332436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-ca\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.332578 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.332493 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-hub\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.344702 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.344676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l25s\" (UniqueName: \"kubernetes.io/projected/06c9660f-273d-4ee7-b345-7ece2f3e4c5f-kube-api-access-6l25s\") pod \"cluster-proxy-proxy-agent-57698d65d8-7nw5b\" (UID: \"06c9660f-273d-4ee7-b345-7ece2f3e4c5f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:08.463093 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:08.463033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" Jul 23 19:17:10.990260 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:10.990189 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b"] Jul 23 19:17:11.255494 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.255406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:17:11.258108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.258078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1db54a43-75dd-4d51-95ac-b95144a6f0fb-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-q9qgs\" (UID: \"1db54a43-75dd-4d51-95ac-b95144a6f0fb\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:17:11.314455 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.314423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" event={"ID":"470c3e2a-e03c-4163-95ca-c268e9544130","Type":"ContainerStarted","Data":"708d654210c40c145026170b6316a1fc52792853d0fff87f871380b5127d672f"} Jul 23 19:17:11.314640 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.314491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" event={"ID":"470c3e2a-e03c-4163-95ca-c268e9544130","Type":"ContainerStarted","Data":"fabfb7ca12f18abc8e476903af79399ece306e473bc512ece285a1ecda6063d8"} Jul 23 19:17:11.316063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.316035 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" event={"ID":"6955e1ef-192b-4f01-9d94-ec04d7e747b6","Type":"ContainerStarted","Data":"5237f32911d47529a44976cf67f929606b9725b13f5a75160e827faa51cc8984"} Jul 23 19:17:11.317854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.317833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n4v72" event={"ID":"21caab7f-0bf4-44bd-8a75-21db01239fb5","Type":"ContainerStarted","Data":"c756f9f902fed70cdf1bb2f7222b6c26980d20f5d0059f3086dce18f6c23bca7"} Jul 23 19:17:11.317948 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.317883 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n4v72" Jul 23 19:17:11.317948 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.317910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n4v72" event={"ID":"21caab7f-0bf4-44bd-8a75-21db01239fb5","Type":"ContainerStarted","Data":"ecfdf16c0ece865355ddd419e620d03508142874f2043ded73a1e658ef21dc4e"} Jul 23 19:17:11.319085 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.319064 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" event={"ID":"06c9660f-273d-4ee7-b345-7ece2f3e4c5f","Type":"ContainerStarted","Data":"7f95601b80f23ef4ed8a429f7b9a7629cd8858a42667ae5ba1e4640e2a204c17"} Jul 23 19:17:11.321047 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.321029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n8l4" event={"ID":"f3869c9d-31a5-437f-96ae-a1f7c54f07cb","Type":"ContainerStarted","Data":"80b52d07da33f63f9a3a77752dc40e2b7cf0f156ceca78c0fa5ddda54957983b"} Jul 23 19:17:11.368793 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.368740 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-j7dt7" podStartSLOduration=33.538125189 podStartE2EDuration="38.368711373s" totalImagesPullingTime="4.830586184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:11.368380543 +0000 UTC m=+78.941001082" watchObservedRunningTime="2026-07-23 19:17:11.368711373 +0000 UTC m=+78.941331913" Jul 23 19:17:11.426199 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.426148 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-kc7v2" podStartSLOduration=33.519165012 podStartE2EDuration="38.426127576s" totalImagesPullingTime="4.906962564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:11.425756161 +0000 UTC m=+78.998376693" watchObservedRunningTime="2026-07-23 19:17:11.426127576 +0000 UTC m=+78.998748119" Jul 23 19:17:11.484262 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.484203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n4v72" podStartSLOduration=38.571994648 podStartE2EDuration="43.484185335s" totalImagesPullingTime="4.912190687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:11.482636883 +0000 UTC m=+79.055257422" watchObservedRunningTime="2026-07-23 19:17:11.484185335 +0000 UTC m=+79.056805853" Jul 23 19:17:11.486250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.486226 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6r77j\"" Jul 23 19:17:11.493715 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.493691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" Jul 23 19:17:11.647754 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:11.647702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs"] Jul 23 19:17:11.650999 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:17:11.650958 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db54a43_75dd_4d51_95ac_b95144a6f0fb.slice/crio-7bc5306ebe0bdf5051b5b3f36863fde51e280ed44bcc48c7c046dd97a98f5d0f WatchSource:0}: Error finding container 7bc5306ebe0bdf5051b5b3f36863fde51e280ed44bcc48c7c046dd97a98f5d0f: Status 404 returned error can't find the container with id 7bc5306ebe0bdf5051b5b3f36863fde51e280ed44bcc48c7c046dd97a98f5d0f Jul 23 19:17:12.329235 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:12.329160 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" event={"ID":"1db54a43-75dd-4d51-95ac-b95144a6f0fb","Type":"ContainerStarted","Data":"7bc5306ebe0bdf5051b5b3f36863fde51e280ed44bcc48c7c046dd97a98f5d0f"} Jul 23 19:17:17.357064 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:17.357016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" event={"ID":"06c9660f-273d-4ee7-b345-7ece2f3e4c5f","Type":"ContainerStarted","Data":"8800d6538b77c2d56c0028e2d64200bb5fde4eff7603bfaa02aacd6afbc7219f"} Jul 23 19:17:17.359199 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:17.359171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6n8l4" event={"ID":"f3869c9d-31a5-437f-96ae-a1f7c54f07cb","Type":"ContainerStarted","Data":"1f24deb6add9df01e6b64c3ce960cb27e79b76e37ebfb55f9ac25ee418a0e7dc"} Jul 23 19:17:17.360827 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:17.360790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" event={"ID":"1db54a43-75dd-4d51-95ac-b95144a6f0fb","Type":"ContainerStarted","Data":"14a0333ab69bce4f989cf878d0ddb2a2feb8e75d8cc7adfa4284424321b1e91e"} Jul 23 19:17:17.385849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:17.385797 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6n8l4" podStartSLOduration=17.403993149 podStartE2EDuration="36.385782414s" totalImagesPullingTime="18.981789265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:16:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:17.384895162 +0000 UTC m=+84.957515702" watchObservedRunningTime="2026-07-23 19:17:17.385782414 +0000 UTC m=+84.958402953" Jul 23 19:17:17.404667 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:17.404624 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-q9qgs" podStartSLOduration=32.819869281 podStartE2EDuration="38.404606604s" totalImagesPullingTime="5.584737323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:17.403802413 +0000 UTC m=+84.976422955" watchObservedRunningTime="2026-07-23 19:17:17.404606604 +0000 UTC m=+84.977227146" Jul 23 19:17:18.366478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:18.366438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" event={"ID":"06c9660f-273d-4ee7-b345-7ece2f3e4c5f","Type":"ContainerStarted","Data":"668ff672229bd5a80f640c7e359cacc147bf78d9cf3fa32df6b5f6f8407321ff"} Jul 23 19:17:18.366882 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:18.366492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" event={"ID":"06c9660f-273d-4ee7-b345-7ece2f3e4c5f","Type":"ContainerStarted","Data":"e9999c32345918131f468cce39fc5e6e19c7ba3ed4270f5e06e5d5a019a785e4"} Jul 23 19:17:18.388061 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:18.388011 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57698d65d8-7nw5b" podStartSLOduration=4.1402064339999995 podStartE2EDuration="10.387995722s" totalImagesPullingTime="6.247789288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:18.386890866 +0000 UTC m=+85.959511400" watchObservedRunningTime="2026-07-23 19:17:18.387995722 +0000 UTC m=+85.960616296" Jul 23 19:17:20.918479 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:20.918446 2574 patch_prober.go:28] interesting pod/image-registry-578c5786db-p5fc7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 19:17:20.918846 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:20.918502 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" podUID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 19:17:21.331971 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:21.331885 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n4v72" Jul 23 19:17:27.303927 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:27.303899 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:28.207554 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.207518 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z4l7b"] Jul 23 19:17:28.212108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.212089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.217319 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.217299 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 19:17:28.217436 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.217318 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bsg5m\"" Jul 23 19:17:28.217756 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.217723 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 19:17:28.217904 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.217766 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 19:17:28.218001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.217784 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 19:17:28.297720 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.297720 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-sys\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.297931 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-metrics-client-ca\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.297931 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-tls\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.297931 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-root\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.298022 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-wtmp\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.298022 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-textfile\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.298022 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.297981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.298022 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.298001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnhj\" (UniqueName: \"kubernetes.io/projected/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-kube-api-access-tbnhj\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.398790 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-root\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-wtmp\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-textfile\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-root\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnhj\" (UniqueName: \"kubernetes.io/projected/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-kube-api-access-tbnhj\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-sys\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-metrics-client-ca\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.398991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-wtmp\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.399027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-tls\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.399204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-sys\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.399228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-textfile\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.399607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-metrics-client-ca\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.399877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.399817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.401404 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.401383 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-tls\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.401525 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.401512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.411949 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.411921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnhj\" (UniqueName: \"kubernetes.io/projected/269dcf6f-ff58-4a89-b4c5-db30038b1dcf-kube-api-access-tbnhj\") pod \"node-exporter-z4l7b\" (UID: \"269dcf6f-ff58-4a89-b4c5-db30038b1dcf\") " pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.521350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:28.521263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4l7b" Jul 23 19:17:28.535341 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:17:28.535314 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269dcf6f_ff58_4a89_b4c5_db30038b1dcf.slice/crio-0c344aac38a379eb7302dab95b21c7f5906b6839d16bea4d09ad0d9de827a1bc WatchSource:0}: Error finding container 0c344aac38a379eb7302dab95b21c7f5906b6839d16bea4d09ad0d9de827a1bc: Status 404 returned error can't find the container with id 0c344aac38a379eb7302dab95b21c7f5906b6839d16bea4d09ad0d9de827a1bc Jul 23 19:17:29.408145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:29.408108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4l7b" event={"ID":"269dcf6f-ff58-4a89-b4c5-db30038b1dcf","Type":"ContainerStarted","Data":"0c344aac38a379eb7302dab95b21c7f5906b6839d16bea4d09ad0d9de827a1bc"} Jul 23 19:17:30.255456 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:30.255424 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x5szb" Jul 23 19:17:30.412901 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:30.412868 2574 generic.go:356] "Generic (PLEG): container finished" podID="269dcf6f-ff58-4a89-b4c5-db30038b1dcf" containerID="082c013aaee896d752b7c8bd1cbc9f5c3101216e6f4c8efd89a1c879635aefce" exitCode=0 Jul 23 19:17:30.413318 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:30.412954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4l7b" event={"ID":"269dcf6f-ff58-4a89-b4c5-db30038b1dcf","Type":"ContainerDied","Data":"082c013aaee896d752b7c8bd1cbc9f5c3101216e6f4c8efd89a1c879635aefce"} Jul 23 19:17:31.418889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:31.418841 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4l7b" event={"ID":"269dcf6f-ff58-4a89-b4c5-db30038b1dcf","Type":"ContainerStarted","Data":"1fc9a93d8b910e614b3221f4db74e3f57a8a1a69814f8867ea2325d95c2f91cf"} Jul 23 19:17:31.419317 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:31.418921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4l7b" event={"ID":"269dcf6f-ff58-4a89-b4c5-db30038b1dcf","Type":"ContainerStarted","Data":"f5d245cdcda5819f33e40316a0d3f5739d87fe32f577f7ee9d50a42c712b8a2b"} Jul 23 19:17:31.441083 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:31.441031 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z4l7b" podStartSLOduration=2.124875312 podStartE2EDuration="3.441014921s" totalImagesPullingTime="1.252863748s" totalInitContainerRuntime="63.275861ms" isStatefulPod=true podCreationTimestamp="2026-07-23 19:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:17:31.439680162 +0000 UTC m=+99.012300693" watchObservedRunningTime="2026-07-23 19:17:31.441014921 +0000 UTC m=+99.013635460" Jul 23 19:17:31.839662 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:31.839577 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-578c5786db-p5fc7"] Jul 23 19:17:51.489222 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:51.489187 2574 generic.go:356] "Generic (PLEG): container finished" podID="4300a1a8-fedf-4191-8cd6-e5818ebbe3cf" containerID="e6ea7175ebbb6d5072c4935cae49eb75b12b479efb96049a45a7fa78ac7f58a3" exitCode=0 Jul 23 19:17:51.489693 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:51.489270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" event={"ID":"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf","Type":"ContainerDied","Data":"e6ea7175ebbb6d5072c4935cae49eb75b12b479efb96049a45a7fa78ac7f58a3"} Jul 23 19:17:51.489693 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:51.489671 2574 scope.go:117] "RemoveContainer" containerID="e6ea7175ebbb6d5072c4935cae49eb75b12b479efb96049a45a7fa78ac7f58a3" Jul 23 19:17:52.497523 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:52.497490 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-n6trx" event={"ID":"4300a1a8-fedf-4191-8cd6-e5818ebbe3cf","Type":"ContainerStarted","Data":"0604fbb145bbab7700b4bf192352553d6080a5e17973c72ae0ba082affb8cab5"} Jul 23 19:17:56.864707 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:56.864658 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" podUID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" containerName="registry" containerID="cri-o://6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5" gracePeriod=30 Jul 23 19:17:57.129968 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.129938 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:57.235406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235373 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e2c84f-1b54-4bed-8c78-5ebf0253c790-ca-trust-extracted\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-certificates\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235642 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235445 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-installation-pull-secrets\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235642 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235642 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235509 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zzmt\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-kube-api-access-9zzmt\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235642 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235571 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-bound-sa-token\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235642 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235614 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-trusted-ca\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-image-registry-private-configuration\") pod \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\" (UID: \"22e2c84f-1b54-4bed-8c78-5ebf0253c790\") " Jul 23 19:17:57.235950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.235848 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:17:57.236220 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.236162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:17:57.236220 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.236187 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-certificates\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.238302 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.238264 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:17:57.238423 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.238367 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-kube-api-access-9zzmt" (OuterVolumeSpecName: "kube-api-access-9zzmt") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "kube-api-access-9zzmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:17:57.238423 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.238379 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:17:57.238511 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.238460 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:17:57.238511 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.238484 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:17:57.244340 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.244317 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e2c84f-1b54-4bed-8c78-5ebf0253c790-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "22e2c84f-1b54-4bed-8c78-5ebf0253c790" (UID: "22e2c84f-1b54-4bed-8c78-5ebf0253c790"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:17:57.337054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337019 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zzmt\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-kube-api-access-9zzmt\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.337054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337048 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-bound-sa-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.337054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337059 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22e2c84f-1b54-4bed-8c78-5ebf0253c790-trusted-ca\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.337281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337069 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-image-registry-private-configuration\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.337281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337078 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22e2c84f-1b54-4bed-8c78-5ebf0253c790-ca-trust-extracted\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.337281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337087 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22e2c84f-1b54-4bed-8c78-5ebf0253c790-installation-pull-secrets\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.337281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.337096 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22e2c84f-1b54-4bed-8c78-5ebf0253c790-registry-tls\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:17:57.517906 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.517869 2574 generic.go:356] "Generic (PLEG): container finished" podID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" containerID="6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5" exitCode=0 Jul 23 19:17:57.518076 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.517918 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" Jul 23 19:17:57.518076 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.517958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" event={"ID":"22e2c84f-1b54-4bed-8c78-5ebf0253c790","Type":"ContainerDied","Data":"6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5"} Jul 23 19:17:57.518076 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.518013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-578c5786db-p5fc7" event={"ID":"22e2c84f-1b54-4bed-8c78-5ebf0253c790","Type":"ContainerDied","Data":"67f0a5093d39d107908cb6b4e8b1bfce734c5c028c575a6944d60c5e65b4127c"} Jul 23 19:17:57.518076 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.518036 2574 scope.go:117] "RemoveContainer" containerID="6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5" Jul 23 19:17:57.526531 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.526514 2574 scope.go:117] "RemoveContainer" containerID="6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5" Jul 23 19:17:57.526809 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:17:57.526789 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5\": container with ID starting with 6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5 not found: ID does not exist" containerID="6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5" Jul 23 19:17:57.526870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.526816 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5"} err="failed to get container status \"6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5\": rpc error: code = NotFound desc = could not find container \"6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5\": container with ID starting with 6b90170bb571215b5b67164e59be3ebb4d994b047e2d99a77b67228b75f5c5b5 not found: ID does not exist" Jul 23 19:17:57.542432 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.542406 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-578c5786db-p5fc7"] Jul 23 19:17:57.549306 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:57.549280 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-578c5786db-p5fc7"] Jul 23 19:17:58.845296 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:17:58.845260 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" path="/var/lib/kubelet/pods/22e2c84f-1b54-4bed-8c78-5ebf0253c790/volumes" Jul 23 19:18:02.538908 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:02.538875 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8b21220-6461-4e91-9760-27f3914bf6d4" containerID="6e135ca7bfb0485a4b85d1b9ddd26772c58cec8835156e954051f27cf443457c" exitCode=0 Jul 23 19:18:02.539270 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:02.538943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" event={"ID":"a8b21220-6461-4e91-9760-27f3914bf6d4","Type":"ContainerDied","Data":"6e135ca7bfb0485a4b85d1b9ddd26772c58cec8835156e954051f27cf443457c"} Jul 23 19:18:02.539334 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:02.539285 2574 scope.go:117] "RemoveContainer" containerID="6e135ca7bfb0485a4b85d1b9ddd26772c58cec8835156e954051f27cf443457c" Jul 23 19:18:03.544453 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:03.544408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-nmxnw" event={"ID":"a8b21220-6461-4e91-9760-27f3914bf6d4","Type":"ContainerStarted","Data":"4ed05bbf27596a28519e227639517f3e46a77db1ec3d803e0653d228a8aa08c7"} Jul 23 19:18:07.562047 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:07.562009 2574 generic.go:356] "Generic (PLEG): container finished" podID="e194a48c-e7dc-495e-a269-dc0a77b33f32" containerID="8dd462ccf07650410f3f3a7d099628dd0fc34132a869ec34c7aaff4d4aeae7eb" exitCode=0 Jul 23 19:18:07.562562 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:07.562084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cr79v" event={"ID":"e194a48c-e7dc-495e-a269-dc0a77b33f32","Type":"ContainerDied","Data":"8dd462ccf07650410f3f3a7d099628dd0fc34132a869ec34c7aaff4d4aeae7eb"} Jul 23 19:18:07.562562 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:07.562425 2574 scope.go:117] "RemoveContainer" containerID="8dd462ccf07650410f3f3a7d099628dd0fc34132a869ec34c7aaff4d4aeae7eb" Jul 23 19:18:08.570029 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:18:08.569995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cr79v" event={"ID":"e194a48c-e7dc-495e-a269-dc0a77b33f32","Type":"ContainerStarted","Data":"b2566d9348152530dda08943229f1453616bdd1fed2d65fef2a4c02035c6be86"} Jul 23 19:20:17.791216 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.791144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m"] Jul 23 19:20:17.791656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.791429 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" containerName="registry" Jul 23 19:20:17.791656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.791441 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" containerName="registry" Jul 23 19:20:17.791656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.791498 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e2c84f-1b54-4bed-8c78-5ebf0253c790" containerName="registry" Jul 23 19:20:17.794368 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.794354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:17.796970 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.796952 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:17.797072 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.796972 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-v9287\"" Jul 23 19:20:17.797285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.797268 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:17.809123 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.809101 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m"] Jul 23 19:20:17.839041 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.839015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f31009d-482c-44f9-a3d5-259b9b015774-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bp48m\" (UID: \"0f31009d-482c-44f9-a3d5-259b9b015774\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:17.839183 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.839070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrbs\" (UniqueName: \"kubernetes.io/projected/0f31009d-482c-44f9-a3d5-259b9b015774-kube-api-access-hcrbs\") pod \"cert-manager-operator-controller-manager-55c444bf86-bp48m\" (UID: \"0f31009d-482c-44f9-a3d5-259b9b015774\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:17.939941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.939905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f31009d-482c-44f9-a3d5-259b9b015774-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bp48m\" (UID: \"0f31009d-482c-44f9-a3d5-259b9b015774\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:17.940074 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.939965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrbs\" (UniqueName: \"kubernetes.io/projected/0f31009d-482c-44f9-a3d5-259b9b015774-kube-api-access-hcrbs\") pod \"cert-manager-operator-controller-manager-55c444bf86-bp48m\" (UID: \"0f31009d-482c-44f9-a3d5-259b9b015774\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:17.940277 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.940258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f31009d-482c-44f9-a3d5-259b9b015774-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bp48m\" (UID: \"0f31009d-482c-44f9-a3d5-259b9b015774\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:17.952921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:17.952898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrbs\" (UniqueName: \"kubernetes.io/projected/0f31009d-482c-44f9-a3d5-259b9b015774-kube-api-access-hcrbs\") pod \"cert-manager-operator-controller-manager-55c444bf86-bp48m\" (UID: \"0f31009d-482c-44f9-a3d5-259b9b015774\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:18.103273 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:18.103190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" Jul 23 19:20:18.239214 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:18.239186 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m"] Jul 23 19:20:18.241933 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:20:18.241909 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f31009d_482c_44f9_a3d5_259b9b015774.slice/crio-afde33582792f278396df8256370bbe98cc28a333153578aded28ec45e430cbb WatchSource:0}: Error finding container afde33582792f278396df8256370bbe98cc28a333153578aded28ec45e430cbb: Status 404 returned error can't find the container with id afde33582792f278396df8256370bbe98cc28a333153578aded28ec45e430cbb Jul 23 19:20:19.042787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:19.042721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" event={"ID":"0f31009d-482c-44f9-a3d5-259b9b015774","Type":"ContainerStarted","Data":"afde33582792f278396df8256370bbe98cc28a333153578aded28ec45e430cbb"} Jul 23 19:20:21.052650 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:21.052610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" event={"ID":"0f31009d-482c-44f9-a3d5-259b9b015774","Type":"ContainerStarted","Data":"3909a54ff13a228ac890db0bff11522cd101139eef8cc2143f415c4b8b52f1a7"} Jul 23 19:20:21.083390 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:21.083336 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bp48m" podStartSLOduration=1.441403569 podStartE2EDuration="4.083318475s" totalImagesPullingTime="2.641914906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:21.082493715 +0000 UTC m=+268.655114278" watchObservedRunningTime="2026-07-23 19:20:21.083318475 +0000 UTC m=+268.655939070" Jul 23 19:20:38.579129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.579095 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k"] Jul 23 19:20:38.582410 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.582392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.585523 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.585504 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 19:20:38.586811 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.586793 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rd7dr\"" Jul 23 19:20:38.586904 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.586842 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 19:20:38.592432 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.592408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k"] Jul 23 19:20:38.710867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.710832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae054ab-1776-4912-aa5f-b326a1b715b6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kcp4k\" (UID: \"3ae054ab-1776-4912-aa5f-b326a1b715b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.711032 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.710889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9lwx\" (UniqueName: \"kubernetes.io/projected/3ae054ab-1776-4912-aa5f-b326a1b715b6-kube-api-access-w9lwx\") pod \"openshift-lws-operator-fd8ccff4c-kcp4k\" (UID: \"3ae054ab-1776-4912-aa5f-b326a1b715b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.812323 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.812269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae054ab-1776-4912-aa5f-b326a1b715b6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kcp4k\" (UID: \"3ae054ab-1776-4912-aa5f-b326a1b715b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.812475 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.812348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9lwx\" (UniqueName: \"kubernetes.io/projected/3ae054ab-1776-4912-aa5f-b326a1b715b6-kube-api-access-w9lwx\") pod \"openshift-lws-operator-fd8ccff4c-kcp4k\" (UID: \"3ae054ab-1776-4912-aa5f-b326a1b715b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.812759 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.812714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae054ab-1776-4912-aa5f-b326a1b715b6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kcp4k\" (UID: \"3ae054ab-1776-4912-aa5f-b326a1b715b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.836220 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.836155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9lwx\" (UniqueName: \"kubernetes.io/projected/3ae054ab-1776-4912-aa5f-b326a1b715b6-kube-api-access-w9lwx\") pod \"openshift-lws-operator-fd8ccff4c-kcp4k\" (UID: \"3ae054ab-1776-4912-aa5f-b326a1b715b6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:38.891347 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:38.891314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" Jul 23 19:20:39.035109 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:39.035082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k"] Jul 23 19:20:39.037588 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:20:39.037560 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae054ab_1776_4912_aa5f_b326a1b715b6.slice/crio-81b11e61b0b4f43f2359dbb3fd587523b684139945bd211fe7a88b85161242c4 WatchSource:0}: Error finding container 81b11e61b0b4f43f2359dbb3fd587523b684139945bd211fe7a88b85161242c4: Status 404 returned error can't find the container with id 81b11e61b0b4f43f2359dbb3fd587523b684139945bd211fe7a88b85161242c4 Jul 23 19:20:39.122673 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:39.122580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" event={"ID":"3ae054ab-1776-4912-aa5f-b326a1b715b6","Type":"ContainerStarted","Data":"81b11e61b0b4f43f2359dbb3fd587523b684139945bd211fe7a88b85161242c4"} Jul 23 19:20:42.143509 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:42.143475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" event={"ID":"3ae054ab-1776-4912-aa5f-b326a1b715b6","Type":"ContainerStarted","Data":"d764504942c1df99a7d8441b27211dd865371f861d6bd24e3cb7b1837067c8cf"} Jul 23 19:20:42.161915 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:42.161862 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kcp4k" podStartSLOduration=1.728736585 podStartE2EDuration="4.161846104s" totalImagesPullingTime="2.433109519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:20:42.160455164 +0000 UTC m=+289.733075715" watchObservedRunningTime="2026-07-23 19:20:42.161846104 +0000 UTC m=+289.734466645" Jul 23 19:20:52.754428 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:52.754387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:20:52.760600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:52.760575 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:20:52.766788 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:52.766760 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 19:20:57.960947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.960913 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7"] Jul 23 19:20:57.967055 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.967029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:57.974267 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.974205 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 19:20:57.974267 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.974238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 19:20:57.975087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.975060 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 19:20:57.975186 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.975151 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hbs2v\"" Jul 23 19:20:57.975625 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.975607 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 19:20:57.985462 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:57.985440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7"] Jul 23 19:20:58.071930 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.071886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ddb66f1-c475-4d2d-a746-3a70f38e6856-tmp\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.072125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.071952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ddb66f1-c475-4d2d-a746-3a70f38e6856-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.072125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.072003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ddb66f1-c475-4d2d-a746-3a70f38e6856-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.072125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.072061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zn4\" (UniqueName: \"kubernetes.io/projected/4ddb66f1-c475-4d2d-a746-3a70f38e6856-kube-api-access-k4zn4\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.072125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.072108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ddb66f1-c475-4d2d-a746-3a70f38e6856-manifests\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.172752 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.172701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zn4\" (UniqueName: \"kubernetes.io/projected/4ddb66f1-c475-4d2d-a746-3a70f38e6856-kube-api-access-k4zn4\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.172951 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.172848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ddb66f1-c475-4d2d-a746-3a70f38e6856-manifests\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.172951 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.172885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ddb66f1-c475-4d2d-a746-3a70f38e6856-tmp\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.172951 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.172919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ddb66f1-c475-4d2d-a746-3a70f38e6856-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.173108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.172961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ddb66f1-c475-4d2d-a746-3a70f38e6856-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.173316 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.173290 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ddb66f1-c475-4d2d-a746-3a70f38e6856-manifests\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.175294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.175276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ddb66f1-c475-4d2d-a746-3a70f38e6856-tmp\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.175360 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.175332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ddb66f1-c475-4d2d-a746-3a70f38e6856-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.175594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.175574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ddb66f1-c475-4d2d-a746-3a70f38e6856-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.182585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.182563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zn4\" (UniqueName: \"kubernetes.io/projected/4ddb66f1-c475-4d2d-a746-3a70f38e6856-kube-api-access-k4zn4\") pod \"opendatahub-operator-controller-manager-6bf4f8c896-cz7h7\" (UID: \"4ddb66f1-c475-4d2d-a746-3a70f38e6856\") " pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.278685 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.278601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:20:58.425824 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.425789 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7"] Jul 23 19:20:58.429593 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:20:58.429566 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb66f1_c475_4d2d_a746_3a70f38e6856.slice/crio-5d821e03590c4e25ed647e7b7e62ffb63f36b12974f325146bdad2cbcd6a9e6b WatchSource:0}: Error finding container 5d821e03590c4e25ed647e7b7e62ffb63f36b12974f325146bdad2cbcd6a9e6b: Status 404 returned error can't find the container with id 5d821e03590c4e25ed647e7b7e62ffb63f36b12974f325146bdad2cbcd6a9e6b Jul 23 19:20:58.431594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:58.431571 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:20:59.214034 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:20:59.213954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" event={"ID":"4ddb66f1-c475-4d2d-a746-3a70f38e6856","Type":"ContainerStarted","Data":"5d821e03590c4e25ed647e7b7e62ffb63f36b12974f325146bdad2cbcd6a9e6b"} Jul 23 19:21:02.228399 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:02.228368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" event={"ID":"4ddb66f1-c475-4d2d-a746-3a70f38e6856","Type":"ContainerStarted","Data":"86217858fea1613b65856861da0756b7a253b4ae1b0be951f8a61139495fbcfc"} Jul 23 19:21:04.239398 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:04.239362 2574 generic.go:356] "Generic (PLEG): container finished" podID="4ddb66f1-c475-4d2d-a746-3a70f38e6856" containerID="86217858fea1613b65856861da0756b7a253b4ae1b0be951f8a61139495fbcfc" exitCode=0 Jul 23 19:21:04.239771 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:04.239437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" event={"ID":"4ddb66f1-c475-4d2d-a746-3a70f38e6856","Type":"ContainerDied","Data":"86217858fea1613b65856861da0756b7a253b4ae1b0be951f8a61139495fbcfc"} Jul 23 19:21:05.245678 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:05.245641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" event={"ID":"4ddb66f1-c475-4d2d-a746-3a70f38e6856","Type":"ContainerStarted","Data":"2803b9bfe3b9ba7e070bee59a1c078fb16ad367026f108e1bf96b6e15f3922de"} Jul 23 19:21:05.246159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:05.245895 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:21:05.290113 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:05.290048 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" podStartSLOduration=3.246857983 podStartE2EDuration="8.290024957s" totalImagesPullingTime="4.022990004s" totalInitContainerRuntime="1.02017697s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:20:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:05.288284409 +0000 UTC m=+312.860904953" watchObservedRunningTime="2026-07-23 19:21:05.290024957 +0000 UTC m=+312.862645497" Jul 23 19:21:16.252428 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:16.252397 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bf4f8c896-cz7h7" Jul 23 19:21:18.636795 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.636760 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k"] Jul 23 19:21:18.642509 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.642486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.645061 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.645033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072438c0-50ff-4360-804e-fa57f5550652-tmp\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.645198 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.645069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgbx\" (UniqueName: \"kubernetes.io/projected/072438c0-50ff-4360-804e-fa57f5550652-kube-api-access-6rgbx\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.645198 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.645099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072438c0-50ff-4360-804e-fa57f5550652-tls-certs\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.645705 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.645686 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 19:21:18.645882 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.645869 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pcf2s\"" Jul 23 19:21:18.646122 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.646109 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 19:21:18.684109 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.684076 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k"] Jul 23 19:21:18.745782 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.745746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072438c0-50ff-4360-804e-fa57f5550652-tmp\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.745782 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.745785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgbx\" (UniqueName: \"kubernetes.io/projected/072438c0-50ff-4360-804e-fa57f5550652-kube-api-access-6rgbx\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.746039 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.745808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072438c0-50ff-4360-804e-fa57f5550652-tls-certs\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.748042 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.748018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/072438c0-50ff-4360-804e-fa57f5550652-tmp\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.748161 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.748143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072438c0-50ff-4360-804e-fa57f5550652-tls-certs\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.767564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.767531 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgbx\" (UniqueName: \"kubernetes.io/projected/072438c0-50ff-4360-804e-fa57f5550652-kube-api-access-6rgbx\") pod \"kube-auth-proxy-6474cf8df7-pcq4k\" (UID: \"072438c0-50ff-4360-804e-fa57f5550652\") " pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:18.919552 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.919516 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vdvc7"] Jul 23 19:21:18.922776 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.922760 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:18.929696 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.929667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t4gtt\"" Jul 23 19:21:18.929860 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.929800 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 19:21:18.947999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.947949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-cert\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:18.948217 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.948022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhrm\" (UniqueName: \"kubernetes.io/projected/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-kube-api-access-bkhrm\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:18.950022 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.949997 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vdvc7"] Jul 23 19:21:18.952321 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:18.952300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" Jul 23 19:21:19.049257 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.049228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-cert\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:19.049406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.049281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhrm\" (UniqueName: \"kubernetes.io/projected/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-kube-api-access-bkhrm\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:19.049406 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:21:19.049382 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 19:21:19.049491 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:21:19.049451 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-cert podName:60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee nodeName:}" failed. No retries permitted until 2026-07-23 19:21:19.549431424 +0000 UTC m=+327.122051957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-cert") pod "odh-model-controller-7998c67dc4-vdvc7" (UID: "60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee") : secret "odh-model-controller-webhook-cert" not found Jul 23 19:21:19.073517 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.073484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhrm\" (UniqueName: \"kubernetes.io/projected/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-kube-api-access-bkhrm\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:19.114761 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.114711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k"] Jul 23 19:21:19.117958 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:21:19.117930 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072438c0_50ff_4360_804e_fa57f5550652.slice/crio-b247770c04584dbcb94e2d9c50de625359ef588ee88acb02d3e09ace95c78c60 WatchSource:0}: Error finding container b247770c04584dbcb94e2d9c50de625359ef588ee88acb02d3e09ace95c78c60: Status 404 returned error can't find the container with id b247770c04584dbcb94e2d9c50de625359ef588ee88acb02d3e09ace95c78c60 Jul 23 19:21:19.305458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.305374 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" event={"ID":"072438c0-50ff-4360-804e-fa57f5550652","Type":"ContainerStarted","Data":"b247770c04584dbcb94e2d9c50de625359ef588ee88acb02d3e09ace95c78c60"} Jul 23 19:21:19.553921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.553885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-cert\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:19.556291 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.556218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee-cert\") pod \"odh-model-controller-7998c67dc4-vdvc7\" (UID: \"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee\") " pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:19.833031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:19.832934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:20.037686 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.037634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vdvc7"] Jul 23 19:21:20.042297 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:21:20.042267 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f9aa80_8dfc_467a_bfdc_ec2d2e6018ee.slice/crio-38a830fa0b37733a2f3ec05d5e1c4bbd36162c37b4fa7bbed209b82be4d1561d WatchSource:0}: Error finding container 38a830fa0b37733a2f3ec05d5e1c4bbd36162c37b4fa7bbed209b82be4d1561d: Status 404 returned error can't find the container with id 38a830fa0b37733a2f3ec05d5e1c4bbd36162c37b4fa7bbed209b82be4d1561d Jul 23 19:21:20.310988 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.310957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" event={"ID":"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee","Type":"ContainerStarted","Data":"38a830fa0b37733a2f3ec05d5e1c4bbd36162c37b4fa7bbed209b82be4d1561d"} Jul 23 19:21:20.655021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.654934 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pzbbn"] Jul 23 19:21:20.659345 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.659328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.663370 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.663346 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 19:21:20.663687 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.663666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8jsk\" (UniqueName: \"kubernetes.io/projected/795b8eaf-8515-4fc4-82cf-dfbfc56b09f1-kube-api-access-h8jsk\") pod \"model-serving-api-54d89bd5fc-pzbbn\" (UID: \"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.663778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.663755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795b8eaf-8515-4fc4-82cf-dfbfc56b09f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzbbn\" (UID: \"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.679964 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.679931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pzbbn"] Jul 23 19:21:20.764849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.764812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795b8eaf-8515-4fc4-82cf-dfbfc56b09f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzbbn\" (UID: \"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.765045 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.764901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8jsk\" (UniqueName: \"kubernetes.io/projected/795b8eaf-8515-4fc4-82cf-dfbfc56b09f1-kube-api-access-h8jsk\") pod \"model-serving-api-54d89bd5fc-pzbbn\" (UID: \"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.767680 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.767656 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795b8eaf-8515-4fc4-82cf-dfbfc56b09f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzbbn\" (UID: \"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.784127 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.784093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8jsk\" (UniqueName: \"kubernetes.io/projected/795b8eaf-8515-4fc4-82cf-dfbfc56b09f1-kube-api-access-h8jsk\") pod \"model-serving-api-54d89bd5fc-pzbbn\" (UID: \"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:20.971410 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:20.971374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:21.180203 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.180174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pzbbn"] Jul 23 19:21:21.319288 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.319192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" event={"ID":"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1","Type":"ContainerStarted","Data":"487f5eaf2e56b8acaa8225c467fc3266048de50cac71da025184479abb17bb61"} Jul 23 19:21:21.793209 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.793171 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z"] Jul 23 19:21:21.796824 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.796802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.803423 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.803400 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 19:21:21.803540 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.803414 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 19:21:21.803540 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.803401 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 19:21:21.803975 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.803952 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 19:21:21.804092 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.804054 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mjhdg\"" Jul 23 19:21:21.820096 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.820063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z"] Jul 23 19:21:21.875765 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-token\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.875945 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.875945 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-cacerts\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.875945 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.875945 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-local-certs\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.875945 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272ph\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-kube-api-access-272ph\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.876169 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.875955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.977519 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272ph\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-kube-api-access-272ph\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-token\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-cacerts\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.977712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-local-certs\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.978323 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.978285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.980691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.980666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-cacerts\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.980885 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.980666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.980968 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.980947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-local-certs\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.981105 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.981018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.993350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.993301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-token\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:21.995381 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:21.995356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-272ph\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-kube-api-access-272ph\") pod \"istiod-openshift-gateway-7574459656-dqt6z\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:22.107958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:22.107862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:23.491840 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:23.491812 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z"] Jul 23 19:21:23.737391 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:21:23.737352 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87c7e7d_9ac1_4418_a6c4_17e7fd9298c4.slice/crio-d2af7af045dd75dae06d7a6930f8a0bbacfa6258f3dde727f2ae7fb9b74756b8 WatchSource:0}: Error finding container d2af7af045dd75dae06d7a6930f8a0bbacfa6258f3dde727f2ae7fb9b74756b8: Status 404 returned error can't find the container with id d2af7af045dd75dae06d7a6930f8a0bbacfa6258f3dde727f2ae7fb9b74756b8 Jul 23 19:21:24.337895 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.337861 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" event={"ID":"60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee","Type":"ContainerStarted","Data":"f416074516167b46e46f9a14841bdbafda66be84e678b13a1cae57e3bd3f511f"} Jul 23 19:21:24.338081 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.338007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:24.339619 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.339595 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" event={"ID":"795b8eaf-8515-4fc4-82cf-dfbfc56b09f1","Type":"ContainerStarted","Data":"99ae52e1717ef84880c04ca210fb4c501407b6b972553a935763ff1981223c7a"} Jul 23 19:21:24.339777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.339642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:24.341327 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.341299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" event={"ID":"072438c0-50ff-4360-804e-fa57f5550652","Type":"ContainerStarted","Data":"ad297fe815422083eda6c9b9ff9ae0ff73c0449ddaaae36c1da95429ed4cc717"} Jul 23 19:21:24.342554 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.342536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" event={"ID":"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4","Type":"ContainerStarted","Data":"d2af7af045dd75dae06d7a6930f8a0bbacfa6258f3dde727f2ae7fb9b74756b8"} Jul 23 19:21:24.359044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.358995 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" podStartSLOduration=2.666093832 podStartE2EDuration="6.358979504s" totalImagesPullingTime="3.692885672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:24.357904722 +0000 UTC m=+331.930525325" watchObservedRunningTime="2026-07-23 19:21:24.358979504 +0000 UTC m=+331.931600044" Jul 23 19:21:24.385268 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.385221 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6474cf8df7-pcq4k" podStartSLOduration=2.191448856 podStartE2EDuration="6.385205937s" totalImagesPullingTime="4.193757081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:24.383622773 +0000 UTC m=+331.956243313" watchObservedRunningTime="2026-07-23 19:21:24.385205937 +0000 UTC m=+331.957826476" Jul 23 19:21:24.405970 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:24.405921 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" podStartSLOduration=1.803610644 podStartE2EDuration="4.405902413s" totalImagesPullingTime="2.602291769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:24.405568536 +0000 UTC m=+331.978189077" watchObservedRunningTime="2026-07-23 19:21:24.405902413 +0000 UTC m=+331.978522958" Jul 23 19:21:26.577991 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:26.577941 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:21:26.578252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:26.578026 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:21:27.360485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:27.360447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" event={"ID":"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4","Type":"ContainerStarted","Data":"065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f"} Jul 23 19:21:27.360696 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:27.360590 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:27.362283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:27.362250 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dqt6z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 19:21:27.362402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:27.362311 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" podUID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 19:21:27.394067 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:27.393999 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" podStartSLOduration=3.555933998 podStartE2EDuration="6.393976002s" totalImagesPullingTime="2.838042004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:27.393106063 +0000 UTC m=+334.965726606" watchObservedRunningTime="2026-07-23 19:21:27.393976002 +0000 UTC m=+334.966596539" Jul 23 19:21:28.350037 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.350001 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd"] Jul 23 19:21:28.353493 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.353473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.356921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.356894 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5db8w\"" Jul 23 19:21:28.365571 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.365549 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:28.376726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.376694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd"] Jul 23 19:21:28.424955 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.424921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425177 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.424975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425177 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93987080-556b-44c8-9956-7e10bb23c22b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425177 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jksp2\" (UniqueName: \"kubernetes.io/projected/93987080-556b-44c8-9956-7e10bb23c22b-kube-api-access-jksp2\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425321 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93987080-556b-44c8-9956-7e10bb23c22b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425321 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425321 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93987080-556b-44c8-9956-7e10bb23c22b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.425525 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.425511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.526826 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.526794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93987080-556b-44c8-9956-7e10bb23c22b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.526999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.526838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jksp2\" (UniqueName: \"kubernetes.io/projected/93987080-556b-44c8-9956-7e10bb23c22b-kube-api-access-jksp2\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.526999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.526859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93987080-556b-44c8-9956-7e10bb23c22b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.526999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.526896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.526999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.526944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93987080-556b-44c8-9956-7e10bb23c22b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527238 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527291 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527342 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527508 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527607 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527671 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527657 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.527769 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.527723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93987080-556b-44c8-9956-7e10bb23c22b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.529285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.529264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93987080-556b-44c8-9956-7e10bb23c22b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.529407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.529388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93987080-556b-44c8-9956-7e10bb23c22b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.540379 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.540345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jksp2\" (UniqueName: \"kubernetes.io/projected/93987080-556b-44c8-9956-7e10bb23c22b-kube-api-access-jksp2\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.542925 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.542902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93987080-556b-44c8-9956-7e10bb23c22b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qjxcd\" (UID: \"93987080-556b-44c8-9956-7e10bb23c22b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.664116 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.664080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:28.882197 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:28.882174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd"] Jul 23 19:21:28.883036 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:21:28.883011 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93987080_556b_44c8_9956_7e10bb23c22b.slice/crio-a816915a56fc4020800001812389dcc354606fcbac798a5dd7eae5b7b8056b17 WatchSource:0}: Error finding container a816915a56fc4020800001812389dcc354606fcbac798a5dd7eae5b7b8056b17: Status 404 returned error can't find the container with id a816915a56fc4020800001812389dcc354606fcbac798a5dd7eae5b7b8056b17 Jul 23 19:21:29.369979 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:29.369942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" event={"ID":"93987080-556b-44c8-9956-7e10bb23c22b","Type":"ContainerStarted","Data":"a816915a56fc4020800001812389dcc354606fcbac798a5dd7eae5b7b8056b17"} Jul 23 19:21:31.098458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.098422 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9622c"] Jul 23 19:21:31.106221 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.106193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.128704 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.128674 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 19:21:31.128897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.128754 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rrxm5\"" Jul 23 19:21:31.145246 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.145213 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9622c"] Jul 23 19:21:31.155913 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.155879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fgj\" (UniqueName: \"kubernetes.io/projected/7f3b62be-cb47-4457-bf01-9159b658eb36-kube-api-access-l7fgj\") pod \"kserve-controller-manager-d6f4b65b8-9622c\" (UID: \"7f3b62be-cb47-4457-bf01-9159b658eb36\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.156095 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.155929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3b62be-cb47-4457-bf01-9159b658eb36-cert\") pod \"kserve-controller-manager-d6f4b65b8-9622c\" (UID: \"7f3b62be-cb47-4457-bf01-9159b658eb36\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.257423 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.257388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fgj\" (UniqueName: \"kubernetes.io/projected/7f3b62be-cb47-4457-bf01-9159b658eb36-kube-api-access-l7fgj\") pod \"kserve-controller-manager-d6f4b65b8-9622c\" (UID: \"7f3b62be-cb47-4457-bf01-9159b658eb36\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.257615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.257432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3b62be-cb47-4457-bf01-9159b658eb36-cert\") pod \"kserve-controller-manager-d6f4b65b8-9622c\" (UID: \"7f3b62be-cb47-4457-bf01-9159b658eb36\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.260222 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.260189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3b62be-cb47-4457-bf01-9159b658eb36-cert\") pod \"kserve-controller-manager-d6f4b65b8-9622c\" (UID: \"7f3b62be-cb47-4457-bf01-9159b658eb36\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.273561 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.273534 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fgj\" (UniqueName: \"kubernetes.io/projected/7f3b62be-cb47-4457-bf01-9159b658eb36-kube-api-access-l7fgj\") pod \"kserve-controller-manager-d6f4b65b8-9622c\" (UID: \"7f3b62be-cb47-4457-bf01-9159b658eb36\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.418789 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.418747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:21:31.788546 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.788518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9622c"] Jul 23 19:21:31.888217 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:21:31.888186 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3b62be_cb47_4457_bf01_9159b658eb36.slice/crio-32460333061b0759d798d3c890b74ba7ce5a6edba05802b6e679ae1a3dd704d5 WatchSource:0}: Error finding container 32460333061b0759d798d3c890b74ba7ce5a6edba05802b6e679ae1a3dd704d5: Status 404 returned error can't find the container with id 32460333061b0759d798d3c890b74ba7ce5a6edba05802b6e679ae1a3dd704d5 Jul 23 19:21:31.919452 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.919408 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:21:31.919581 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.919496 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:21:31.919581 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.919540 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:21:31.921895 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.921872 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z"] Jul 23 19:21:31.940438 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:31.940361 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" podUID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" containerName="discovery" containerID="cri-o://065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f" gracePeriod=30 Jul 23 19:21:32.223995 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.223952 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:32.269030 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.268999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-local-certs\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.269150 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269109 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-cacerts\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.269150 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269137 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-dns-cert\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.269254 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269194 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-token\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.269254 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269223 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-kubeconfig\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.269353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269253 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-ca-configmap\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.269353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269281 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-272ph\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-kube-api-access-272ph\") pod \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\" (UID: \"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4\") " Jul 23 19:21:32.270137 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.269862 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:21:32.271924 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.271894 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-local-certs" (OuterVolumeSpecName: "local-certs") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:21:32.272038 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.271963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:21:32.272327 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.272307 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-kube-api-access-272ph" (OuterVolumeSpecName: "kube-api-access-272ph") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "kube-api-access-272ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:21:32.272811 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.272789 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-cacerts" (OuterVolumeSpecName: "cacerts") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:21:32.273213 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.273190 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:21:32.273360 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.273331 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" (UID: "a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:21:32.370637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370597 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-cacerts\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.370637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370630 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-dns-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.370637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370640 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.370897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370650 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-kubeconfig\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.370897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370658 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-istio-csr-ca-configmap\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.370897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370667 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-272ph\" (UniqueName: \"kubernetes.io/projected/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-kube-api-access-272ph\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.370897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.370675 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4-local-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:21:32.386042 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.386010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" event={"ID":"7f3b62be-cb47-4457-bf01-9159b658eb36","Type":"ContainerStarted","Data":"32460333061b0759d798d3c890b74ba7ce5a6edba05802b6e679ae1a3dd704d5"} Jul 23 19:21:32.387239 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.387205 2574 generic.go:356] "Generic (PLEG): container finished" podID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" containerID="065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f" exitCode=0 Jul 23 19:21:32.387389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.387253 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" Jul 23 19:21:32.387389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.387288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" event={"ID":"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4","Type":"ContainerDied","Data":"065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f"} Jul 23 19:21:32.387389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.387330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z" event={"ID":"a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4","Type":"ContainerDied","Data":"d2af7af045dd75dae06d7a6930f8a0bbacfa6258f3dde727f2ae7fb9b74756b8"} Jul 23 19:21:32.387389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.387357 2574 scope.go:117] "RemoveContainer" containerID="065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f" Jul 23 19:21:32.388897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.388859 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" event={"ID":"93987080-556b-44c8-9956-7e10bb23c22b","Type":"ContainerStarted","Data":"2378da9a85da24f45dbd8deeb0893d547c7c0c98dfb72a3b667f1fd97a033d0d"} Jul 23 19:21:32.396438 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.396419 2574 scope.go:117] "RemoveContainer" containerID="065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f" Jul 23 19:21:32.396791 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:21:32.396769 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f\": container with ID starting with 065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f not found: ID does not exist" containerID="065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f" Jul 23 19:21:32.396843 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.396801 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f"} err="failed to get container status \"065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f\": rpc error: code = NotFound desc = could not find container \"065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f\": container with ID starting with 065737b6160e02f7cd30a68f53f31fd479a42a5c4ad11b486a6a10b9f2ff7f8f not found: ID does not exist" Jul 23 19:21:32.436775 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.436741 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z"] Jul 23 19:21:32.447754 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.447655 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dqt6z"] Jul 23 19:21:32.562986 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.562940 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" podStartSLOduration=1.528905435 podStartE2EDuration="4.562922161s" totalImagesPullingTime="3.034016726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:32.558687367 +0000 UTC m=+340.131307918" watchObservedRunningTime="2026-07-23 19:21:32.562922161 +0000 UTC m=+340.135542700" Jul 23 19:21:32.664330 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.664290 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:32.847202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:32.847119 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" path="/var/lib/kubelet/pods/a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4/volumes" Jul 23 19:21:33.670251 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:33.670221 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:34.399630 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:34.399597 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:34.400788 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:34.400763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qjxcd" Jul 23 19:21:35.351999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:35.351970 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vdvc7" Jul 23 19:21:35.353881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:35.353862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pzbbn" Jul 23 19:21:35.405692 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:35.405655 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" event={"ID":"7f3b62be-cb47-4457-bf01-9159b658eb36","Type":"ContainerStarted","Data":"1a737a5682316cd90abe6a75f9e33aab33c2aa860262bfaccd31564e8e4497f5"} Jul 23 19:21:36.410652 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:21:36.410625 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:22:07.419535 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:07.419499 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" Jul 23 19:22:07.452318 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:07.452263 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9622c" podStartSLOduration=33.816951387 podStartE2EDuration="36.452246635s" totalImagesPullingTime="2.635295248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:21:35.425985988 +0000 UTC m=+342.998606525" watchObservedRunningTime="2026-07-23 19:22:07.452246635 +0000 UTC m=+375.024867153" Jul 23 19:22:19.676351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.676320 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmrzv"] Jul 23 19:22:19.676726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.676631 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" containerName="discovery" Jul 23 19:22:19.676726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.676642 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" containerName="discovery" Jul 23 19:22:19.676726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.676700 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c7e7d-9ac1-4418-a6c4-17e7fd9298c4" containerName="discovery" Jul 23 19:22:19.679718 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.679703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:19.683279 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.683251 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-2tg7l\"" Jul 23 19:22:19.683422 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.683252 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 19:22:19.683780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.683759 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 19:22:19.714100 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.714068 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmrzv"] Jul 23 19:22:19.762221 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.762186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdkm\" (UniqueName: \"kubernetes.io/projected/519ff09b-d0cc-442c-8a07-25d0bfb1f86e-kube-api-access-sxdkm\") pod \"kuadrant-operator-catalog-wmrzv\" (UID: \"519ff09b-d0cc-442c-8a07-25d0bfb1f86e\") " pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:19.862877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.862844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdkm\" (UniqueName: \"kubernetes.io/projected/519ff09b-d0cc-442c-8a07-25d0bfb1f86e-kube-api-access-sxdkm\") pod \"kuadrant-operator-catalog-wmrzv\" (UID: \"519ff09b-d0cc-442c-8a07-25d0bfb1f86e\") " pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:19.881620 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.881583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdkm\" (UniqueName: \"kubernetes.io/projected/519ff09b-d0cc-442c-8a07-25d0bfb1f86e-kube-api-access-sxdkm\") pod \"kuadrant-operator-catalog-wmrzv\" (UID: \"519ff09b-d0cc-442c-8a07-25d0bfb1f86e\") " pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:19.979498 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.979404 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmrzv"] Jul 23 19:22:19.979680 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:19.979666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:20.122123 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.122048 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmrzv"] Jul 23 19:22:20.124423 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:22:20.124399 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519ff09b_d0cc_442c_8a07_25d0bfb1f86e.slice/crio-9c7193d0919d6c5c4752ff0627215c7e2f5e5f254eb39dcecd381342739d5876 WatchSource:0}: Error finding container 9c7193d0919d6c5c4752ff0627215c7e2f5e5f254eb39dcecd381342739d5876: Status 404 returned error can't find the container with id 9c7193d0919d6c5c4752ff0627215c7e2f5e5f254eb39dcecd381342739d5876 Jul 23 19:22:20.199515 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.199481 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59nfn"] Jul 23 19:22:20.203961 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.203944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:20.218723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.218693 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59nfn"] Jul 23 19:22:20.265216 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.265142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54rr\" (UniqueName: \"kubernetes.io/projected/bc384ab1-501c-49b4-9dd6-8fa2b57f7e06-kube-api-access-c54rr\") pod \"kuadrant-operator-catalog-59nfn\" (UID: \"bc384ab1-501c-49b4-9dd6-8fa2b57f7e06\") " pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:20.365662 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.365624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54rr\" (UniqueName: \"kubernetes.io/projected/bc384ab1-501c-49b4-9dd6-8fa2b57f7e06-kube-api-access-c54rr\") pod \"kuadrant-operator-catalog-59nfn\" (UID: \"bc384ab1-501c-49b4-9dd6-8fa2b57f7e06\") " pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:20.376592 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.376550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54rr\" (UniqueName: \"kubernetes.io/projected/bc384ab1-501c-49b4-9dd6-8fa2b57f7e06-kube-api-access-c54rr\") pod \"kuadrant-operator-catalog-59nfn\" (UID: \"bc384ab1-501c-49b4-9dd6-8fa2b57f7e06\") " pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:20.512654 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.512622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:20.613915 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.613879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" event={"ID":"519ff09b-d0cc-442c-8a07-25d0bfb1f86e","Type":"ContainerStarted","Data":"9c7193d0919d6c5c4752ff0627215c7e2f5e5f254eb39dcecd381342739d5876"} Jul 23 19:22:20.655757 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:20.655687 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-59nfn"] Jul 23 19:22:20.657967 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:22:20.657940 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc384ab1_501c_49b4_9dd6_8fa2b57f7e06.slice/crio-a1a3bef0a13a6a0abcc71811fffb5008bc1fc2b01cbb6e55a1451b7d38495e8f WatchSource:0}: Error finding container a1a3bef0a13a6a0abcc71811fffb5008bc1fc2b01cbb6e55a1451b7d38495e8f: Status 404 returned error can't find the container with id a1a3bef0a13a6a0abcc71811fffb5008bc1fc2b01cbb6e55a1451b7d38495e8f Jul 23 19:22:21.621171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:21.621135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" event={"ID":"bc384ab1-501c-49b4-9dd6-8fa2b57f7e06","Type":"ContainerStarted","Data":"a1a3bef0a13a6a0abcc71811fffb5008bc1fc2b01cbb6e55a1451b7d38495e8f"} Jul 23 19:22:23.631801 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.631757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" event={"ID":"519ff09b-d0cc-442c-8a07-25d0bfb1f86e","Type":"ContainerStarted","Data":"1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe"} Jul 23 19:22:23.632288 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.631786 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" podUID="519ff09b-d0cc-442c-8a07-25d0bfb1f86e" containerName="registry-server" containerID="cri-o://1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe" gracePeriod=2 Jul 23 19:22:23.633328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.633298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" event={"ID":"bc384ab1-501c-49b4-9dd6-8fa2b57f7e06","Type":"ContainerStarted","Data":"a0d237db51a6a26e09710a8ac05424f8e633318480d6080db50e5418cfa94ef6"} Jul 23 19:22:23.648493 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.648447 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" podStartSLOduration=2.002589913 podStartE2EDuration="4.648434154s" totalImagesPullingTime="2.645844241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:22:23.647293725 +0000 UTC m=+391.219914265" watchObservedRunningTime="2026-07-23 19:22:23.648434154 +0000 UTC m=+391.221054693" Jul 23 19:22:23.664593 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.664544 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" podStartSLOduration=1.54935574 podStartE2EDuration="3.664529347s" totalImagesPullingTime="2.115173607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:22:23.663438013 +0000 UTC m=+391.236058555" watchObservedRunningTime="2026-07-23 19:22:23.664529347 +0000 UTC m=+391.237149887" Jul 23 19:22:23.880369 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.880343 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:23.896489 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.896438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxdkm\" (UniqueName: \"kubernetes.io/projected/519ff09b-d0cc-442c-8a07-25d0bfb1f86e-kube-api-access-sxdkm\") pod \"519ff09b-d0cc-442c-8a07-25d0bfb1f86e\" (UID: \"519ff09b-d0cc-442c-8a07-25d0bfb1f86e\") " Jul 23 19:22:23.898510 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.898478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519ff09b-d0cc-442c-8a07-25d0bfb1f86e-kube-api-access-sxdkm" (OuterVolumeSpecName: "kube-api-access-sxdkm") pod "519ff09b-d0cc-442c-8a07-25d0bfb1f86e" (UID: "519ff09b-d0cc-442c-8a07-25d0bfb1f86e"). InnerVolumeSpecName "kube-api-access-sxdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:22:23.997281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:23.997248 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdkm\" (UniqueName: \"kubernetes.io/projected/519ff09b-d0cc-442c-8a07-25d0bfb1f86e-kube-api-access-sxdkm\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:22:24.638783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.638723 2574 generic.go:356] "Generic (PLEG): container finished" podID="519ff09b-d0cc-442c-8a07-25d0bfb1f86e" containerID="1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe" exitCode=0 Jul 23 19:22:24.638783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.638778 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" Jul 23 19:22:24.639211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.638825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" event={"ID":"519ff09b-d0cc-442c-8a07-25d0bfb1f86e","Type":"ContainerDied","Data":"1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe"} Jul 23 19:22:24.639211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.638866 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wmrzv" event={"ID":"519ff09b-d0cc-442c-8a07-25d0bfb1f86e","Type":"ContainerDied","Data":"9c7193d0919d6c5c4752ff0627215c7e2f5e5f254eb39dcecd381342739d5876"} Jul 23 19:22:24.639211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.638889 2574 scope.go:117] "RemoveContainer" containerID="1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe" Jul 23 19:22:24.647664 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.647643 2574 scope.go:117] "RemoveContainer" containerID="1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe" Jul 23 19:22:24.648012 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:22:24.647993 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe\": container with ID starting with 1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe not found: ID does not exist" containerID="1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe" Jul 23 19:22:24.648063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.648020 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe"} err="failed to get container status \"1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe\": rpc error: code = NotFound desc = could not find container \"1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe\": container with ID starting with 1e2a1cb262b2a27f97fb38ffb99bab487c8528c69ed631914434fd2361453cfe not found: ID does not exist" Jul 23 19:22:24.665325 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.665295 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmrzv"] Jul 23 19:22:24.668773 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.668747 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wmrzv"] Jul 23 19:22:24.846812 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:24.846783 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519ff09b-d0cc-442c-8a07-25d0bfb1f86e" path="/var/lib/kubelet/pods/519ff09b-d0cc-442c-8a07-25d0bfb1f86e/volumes" Jul 23 19:22:30.513384 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:30.513342 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:30.513842 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:30.513401 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:30.535856 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:30.535831 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:30.689382 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:30.689354 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-59nfn" Jul 23 19:22:53.593077 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.593038 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn"] Jul 23 19:22:53.593571 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.593523 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519ff09b-d0cc-442c-8a07-25d0bfb1f86e" containerName="registry-server" Jul 23 19:22:53.593571 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.593545 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="519ff09b-d0cc-442c-8a07-25d0bfb1f86e" containerName="registry-server" Jul 23 19:22:53.593688 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.593656 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="519ff09b-d0cc-442c-8a07-25d0bfb1f86e" containerName="registry-server" Jul 23 19:22:53.596968 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.596947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.603601 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.603394 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-shfjv\"" Jul 23 19:22:53.610825 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.610796 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn"] Jul 23 19:22:53.756608 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.756566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16b3d-8caf-465b-9f58-e34a894c3583-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.756855 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.756633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvf5\" (UniqueName: \"kubernetes.io/projected/b6f16b3d-8caf-465b-9f58-e34a894c3583-kube-api-access-klvf5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.857938 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.857849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16b3d-8caf-465b-9f58-e34a894c3583-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.857938 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.857901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvf5\" (UniqueName: \"kubernetes.io/projected/b6f16b3d-8caf-465b-9f58-e34a894c3583-kube-api-access-klvf5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.858212 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.858192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16b3d-8caf-465b-9f58-e34a894c3583-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.876922 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.876895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvf5\" (UniqueName: \"kubernetes.io/projected/b6f16b3d-8caf-465b-9f58-e34a894c3583-kube-api-access-klvf5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:53.906873 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:53.906845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:22:54.066689 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:54.066665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn"] Jul 23 19:22:54.068724 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:22:54.068699 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f16b3d_8caf_465b_9f58_e34a894c3583.slice/crio-3807703dfa6935a7e5a62d1e0c1d219f4b165e17d7e4780f57afa5f1aea2ea12 WatchSource:0}: Error finding container 3807703dfa6935a7e5a62d1e0c1d219f4b165e17d7e4780f57afa5f1aea2ea12: Status 404 returned error can't find the container with id 3807703dfa6935a7e5a62d1e0c1d219f4b165e17d7e4780f57afa5f1aea2ea12 Jul 23 19:22:54.778810 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:54.778769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" event={"ID":"b6f16b3d-8caf-465b-9f58-e34a894c3583","Type":"ContainerStarted","Data":"3807703dfa6935a7e5a62d1e0c1d219f4b165e17d7e4780f57afa5f1aea2ea12"} Jul 23 19:22:55.527489 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.527445 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq"] Jul 23 19:22:55.535288 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.535261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:22:55.539942 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.539913 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6v7kr\"" Jul 23 19:22:55.544862 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.544837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq"] Jul 23 19:22:55.672658 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.672623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvvj\" (UniqueName: \"kubernetes.io/projected/04973124-1eb3-437d-b40c-e82f0e2e39e9-kube-api-access-4pvvj\") pod \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" (UID: \"04973124-1eb3-437d-b40c-e82f0e2e39e9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:22:55.773509 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.773466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvvj\" (UniqueName: \"kubernetes.io/projected/04973124-1eb3-437d-b40c-e82f0e2e39e9-kube-api-access-4pvvj\") pod \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" (UID: \"04973124-1eb3-437d-b40c-e82f0e2e39e9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:22:55.788158 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.788089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvvj\" (UniqueName: \"kubernetes.io/projected/04973124-1eb3-437d-b40c-e82f0e2e39e9-kube-api-access-4pvvj\") pod \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" (UID: \"04973124-1eb3-437d-b40c-e82f0e2e39e9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:22:55.847159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:55.847127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:22:56.034168 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:56.034142 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq"] Jul 23 19:22:56.036067 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:22:56.036038 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04973124_1eb3_437d_b40c_e82f0e2e39e9.slice/crio-ea89f684cfce232e1179a25f0c475d39e0e43cfcbc00acaadfbb1ecd0ccc1a31 WatchSource:0}: Error finding container ea89f684cfce232e1179a25f0c475d39e0e43cfcbc00acaadfbb1ecd0ccc1a31: Status 404 returned error can't find the container with id ea89f684cfce232e1179a25f0c475d39e0e43cfcbc00acaadfbb1ecd0ccc1a31 Jul 23 19:22:56.791400 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:56.791357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" event={"ID":"04973124-1eb3-437d-b40c-e82f0e2e39e9","Type":"ContainerStarted","Data":"ea89f684cfce232e1179a25f0c475d39e0e43cfcbc00acaadfbb1ecd0ccc1a31"} Jul 23 19:22:57.972597 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:57.972498 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d"] Jul 23 19:22:57.976302 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:57.976280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:22:57.980636 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:57.980618 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-h2tf8\"" Jul 23 19:22:57.987538 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:57.987513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d"] Jul 23 19:22:58.093885 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.093851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvg94\" (UniqueName: \"kubernetes.io/projected/2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307-kube-api-access-tvg94\") pod \"authorino-operator-75cc8d5cb7-cxt9d\" (UID: \"2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:22:58.194846 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.194806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvg94\" (UniqueName: \"kubernetes.io/projected/2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307-kube-api-access-tvg94\") pod \"authorino-operator-75cc8d5cb7-cxt9d\" (UID: \"2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:22:58.208688 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.208561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvg94\" (UniqueName: \"kubernetes.io/projected/2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307-kube-api-access-tvg94\") pod \"authorino-operator-75cc8d5cb7-cxt9d\" (UID: \"2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:22:58.285794 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.285706 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:22:58.435599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.435513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d"] Jul 23 19:22:58.438351 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:22:58.438321 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf4bb0_d5fe_4fd6_8ab0_d7f15f1dd307.slice/crio-a86263a27d6453f854bbed53da1d19d6c934070dbe92de02e9a7c13963b96b7a WatchSource:0}: Error finding container a86263a27d6453f854bbed53da1d19d6c934070dbe92de02e9a7c13963b96b7a: Status 404 returned error can't find the container with id a86263a27d6453f854bbed53da1d19d6c934070dbe92de02e9a7c13963b96b7a Jul 23 19:22:58.804123 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.804085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" event={"ID":"2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307","Type":"ContainerStarted","Data":"a86263a27d6453f854bbed53da1d19d6c934070dbe92de02e9a7c13963b96b7a"} Jul 23 19:22:58.805980 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.805953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" event={"ID":"04973124-1eb3-437d-b40c-e82f0e2e39e9","Type":"ContainerStarted","Data":"629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb"} Jul 23 19:22:58.806302 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.806279 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:22:58.824988 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:22:58.824939 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" podStartSLOduration=2.138727676 podStartE2EDuration="3.824921324s" totalImagesPullingTime="1.686193648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:22:58.824776537 +0000 UTC m=+426.397397082" watchObservedRunningTime="2026-07-23 19:22:58.824921324 +0000 UTC m=+426.397541865" Jul 23 19:23:00.819818 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:00.819784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" event={"ID":"b6f16b3d-8caf-465b-9f58-e34a894c3583","Type":"ContainerStarted","Data":"1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9"} Jul 23 19:23:00.820215 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:00.819935 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:00.842454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:00.842388 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" podStartSLOduration=1.9535315519999998 podStartE2EDuration="7.842365523s" totalImagesPullingTime="5.888833971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:00.839789774 +0000 UTC m=+428.412410318" watchObservedRunningTime="2026-07-23 19:23:00.842365523 +0000 UTC m=+428.414986065" Jul 23 19:23:02.830691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:02.830653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" event={"ID":"2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307","Type":"ContainerStarted","Data":"4901f0d011763f480b1f83b185b3f4829493ad01a64f08c536a49dda51956554"} Jul 23 19:23:02.831136 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:02.830783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:23:02.877462 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:02.877406 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" podStartSLOduration=2.5461322170000003 podStartE2EDuration="5.87739088s" totalImagesPullingTime="3.331258663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:02.87734268 +0000 UTC m=+430.449963220" watchObservedRunningTime="2026-07-23 19:23:02.87739088 +0000 UTC m=+430.450011419" Jul 23 19:23:03.277967 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.277927 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69"] Jul 23 19:23:03.281656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.281627 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.285120 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.285095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 23 19:23:03.285698 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.285679 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 23 19:23:03.285894 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.285803 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-674zl\"" Jul 23 19:23:03.301150 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.301120 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69"] Jul 23 19:23:03.439683 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.439646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.439874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.439706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq8h\" (UniqueName: \"kubernetes.io/projected/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-kube-api-access-6jq8h\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.439874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.439829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.541102 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.541009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq8h\" (UniqueName: \"kubernetes.io/projected/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-kube-api-access-6jq8h\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.541102 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.541059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.541285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.541132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.541870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.541848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.543600 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.543568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.551968 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.551939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq8h\" (UniqueName: \"kubernetes.io/projected/a0d629ce-0a9c-48c6-accd-87ab1e5e7685-kube-api-access-6jq8h\") pod \"kuadrant-console-plugin-66b5774985-tgd69\" (UID: \"a0d629ce-0a9c-48c6-accd-87ab1e5e7685\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.591751 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.591703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" Jul 23 19:23:03.732939 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.732915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69"] Jul 23 19:23:03.734797 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:23:03.734768 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d629ce_0a9c_48c6_accd_87ab1e5e7685.slice/crio-79c28eba5aad74ebfe0e6c2acbaf4a0ef5e8ea94e8f770f45f4f431211bc9919 WatchSource:0}: Error finding container 79c28eba5aad74ebfe0e6c2acbaf4a0ef5e8ea94e8f770f45f4f431211bc9919: Status 404 returned error can't find the container with id 79c28eba5aad74ebfe0e6c2acbaf4a0ef5e8ea94e8f770f45f4f431211bc9919 Jul 23 19:23:03.836864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:03.836763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" event={"ID":"a0d629ce-0a9c-48c6-accd-87ab1e5e7685","Type":"ContainerStarted","Data":"79c28eba5aad74ebfe0e6c2acbaf4a0ef5e8ea94e8f770f45f4f431211bc9919"} Jul 23 19:23:09.815246 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:09.815212 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:11.826896 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:11.826865 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:13.517688 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.517651 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn"] Jul 23 19:23:13.518155 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.517915 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" containerName="manager" containerID="cri-o://1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9" gracePeriod=2 Jul 23 19:23:13.526243 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.526204 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn"] Jul 23 19:23:13.538719 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.538686 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq"] Jul 23 19:23:13.538979 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.538928 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" containerName="manager" containerID="cri-o://629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb" gracePeriod=2 Jul 23 19:23:13.548936 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.548766 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:13.552043 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.552012 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8"] Jul 23 19:23:13.552439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.552413 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" containerName="manager" Jul 23 19:23:13.552439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.552433 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" containerName="manager" Jul 23 19:23:13.552626 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.552525 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" containerName="manager" Jul 23 19:23:13.557555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.557528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.562756 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.562712 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq"] Jul 23 19:23:13.582100 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.581855 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"04973124-1eb3-437d-b40c-e82f0e2e39e9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-kf9xq\": pods \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:13.583163 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.583043 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8"] Jul 23 19:23:13.584503 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.584401 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:13.586578 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.586554 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj"] Jul 23 19:23:13.587078 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.587061 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" containerName="manager" Jul 23 19:23:13.587078 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.587079 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" containerName="manager" Jul 23 19:23:13.587191 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.587169 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" containerName="manager" Jul 23 19:23:13.588102 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.588070 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:13.593435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.593408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:13.616962 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.616930 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj"] Jul 23 19:23:13.635459 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.635411 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:13.637499 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.637456 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:13.646816 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.646792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e8f1633-d9a6-4edc-8336-85457abb39e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2src8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.646983 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.646961 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfkj\" (UniqueName: \"kubernetes.io/projected/4e8f1633-d9a6-4edc-8336-85457abb39e8-kube-api-access-pwfkj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2src8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.647047 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.647035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkh7\" (UniqueName: \"kubernetes.io/projected/b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e-kube-api-access-5lkh7\") pod \"limitador-operator-controller-manager-859d7b78bd-8zwxj\" (UID: \"b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:13.748449 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.748411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkh7\" (UniqueName: \"kubernetes.io/projected/b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e-kube-api-access-5lkh7\") pod \"limitador-operator-controller-manager-859d7b78bd-8zwxj\" (UID: \"b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:13.748629 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.748499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e8f1633-d9a6-4edc-8336-85457abb39e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2src8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.748629 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.748540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfkj\" (UniqueName: \"kubernetes.io/projected/4e8f1633-d9a6-4edc-8336-85457abb39e8-kube-api-access-pwfkj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2src8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.748955 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.748933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e8f1633-d9a6-4edc-8336-85457abb39e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2src8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.768872 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.768790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfkj\" (UniqueName: \"kubernetes.io/projected/4e8f1633-d9a6-4edc-8336-85457abb39e8-kube-api-access-pwfkj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2src8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.768872 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.768849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkh7\" (UniqueName: \"kubernetes.io/projected/b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e-kube-api-access-5lkh7\") pod \"limitador-operator-controller-manager-859d7b78bd-8zwxj\" (UID: \"b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:13.839304 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.839270 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cxt9d" Jul 23 19:23:13.841798 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.841762 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:13.871078 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:13.871034 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:13.946797 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.946749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:13.954378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:13.954354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:26.441439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.441409 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:26.443800 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:26.443774 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:26.444099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.444078 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:26.448421 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.448399 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8"] Jul 23 19:23:26.449044 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:26.449011 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:26.451018 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:23:26.450967 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8f1633_d9a6_4edc_8336_85457abb39e8.slice/crio-9dd618b3fd068baec1e16d283f12b4a29b90dd6dd9b662a6fd023176e9315d00 WatchSource:0}: Error finding container 9dd618b3fd068baec1e16d283f12b4a29b90dd6dd9b662a6fd023176e9315d00: Status 404 returned error can't find the container with id 9dd618b3fd068baec1e16d283f12b4a29b90dd6dd9b662a6fd023176e9315d00 Jul 23 19:23:26.451128 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:26.451052 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:26.471919 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.471897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj"] Jul 23 19:23:26.485880 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:23:26.485856 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b9f32d_53c1_4a1c_96df_f51b1d3ed34e.slice/crio-8367b14de68637bd24c33e89d879e5e8c80cb9b3c199b816d027a2162baffa2c WatchSource:0}: Error finding container 8367b14de68637bd24c33e89d879e5e8c80cb9b3c199b816d027a2162baffa2c: Status 404 returned error can't find the container with id 8367b14de68637bd24c33e89d879e5e8c80cb9b3c199b816d027a2162baffa2c Jul 23 19:23:26.568609 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.568575 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klvf5\" (UniqueName: \"kubernetes.io/projected/b6f16b3d-8caf-465b-9f58-e34a894c3583-kube-api-access-klvf5\") pod \"b6f16b3d-8caf-465b-9f58-e34a894c3583\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " Jul 23 19:23:26.568783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.568626 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16b3d-8caf-465b-9f58-e34a894c3583-extensions-socket-volume\") pod \"b6f16b3d-8caf-465b-9f58-e34a894c3583\" (UID: \"b6f16b3d-8caf-465b-9f58-e34a894c3583\") " Jul 23 19:23:26.568783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.568664 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pvvj\" (UniqueName: \"kubernetes.io/projected/04973124-1eb3-437d-b40c-e82f0e2e39e9-kube-api-access-4pvvj\") pod \"04973124-1eb3-437d-b40c-e82f0e2e39e9\" (UID: \"04973124-1eb3-437d-b40c-e82f0e2e39e9\") " Jul 23 19:23:26.569196 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.569169 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f16b3d-8caf-465b-9f58-e34a894c3583-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b6f16b3d-8caf-465b-9f58-e34a894c3583" (UID: "b6f16b3d-8caf-465b-9f58-e34a894c3583"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:23:26.570693 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.570666 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04973124-1eb3-437d-b40c-e82f0e2e39e9-kube-api-access-4pvvj" (OuterVolumeSpecName: "kube-api-access-4pvvj") pod "04973124-1eb3-437d-b40c-e82f0e2e39e9" (UID: "04973124-1eb3-437d-b40c-e82f0e2e39e9"). InnerVolumeSpecName "kube-api-access-4pvvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:23:26.570804 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.570711 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f16b3d-8caf-465b-9f58-e34a894c3583-kube-api-access-klvf5" (OuterVolumeSpecName: "kube-api-access-klvf5") pod "b6f16b3d-8caf-465b-9f58-e34a894c3583" (UID: "b6f16b3d-8caf-465b-9f58-e34a894c3583"). InnerVolumeSpecName "kube-api-access-klvf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:23:26.670144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.670108 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klvf5\" (UniqueName: \"kubernetes.io/projected/b6f16b3d-8caf-465b-9f58-e34a894c3583-kube-api-access-klvf5\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:23:26.670144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.670137 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6f16b3d-8caf-465b-9f58-e34a894c3583-extensions-socket-volume\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:23:26.670144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.670147 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pvvj\" (UniqueName: \"kubernetes.io/projected/04973124-1eb3-437d-b40c-e82f0e2e39e9-kube-api-access-4pvvj\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:23:26.847169 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.847088 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" path="/var/lib/kubelet/pods/04973124-1eb3-437d-b40c-e82f0e2e39e9/volumes" Jul 23 19:23:26.847446 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.847433 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" path="/var/lib/kubelet/pods/b6f16b3d-8caf-465b-9f58-e34a894c3583/volumes" Jul 23 19:23:26.964534 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.964494 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" event={"ID":"4e8f1633-d9a6-4edc-8336-85457abb39e8","Type":"ContainerStarted","Data":"d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e"} Jul 23 19:23:26.964761 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.964556 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" event={"ID":"4e8f1633-d9a6-4edc-8336-85457abb39e8","Type":"ContainerStarted","Data":"9dd618b3fd068baec1e16d283f12b4a29b90dd6dd9b662a6fd023176e9315d00"} Jul 23 19:23:26.964761 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.964582 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:26.965708 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.965682 2574 generic.go:356] "Generic (PLEG): container finished" podID="b6f16b3d-8caf-465b-9f58-e34a894c3583" containerID="1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9" exitCode=0 Jul 23 19:23:26.965849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.965720 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:26.965849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.965788 2574 scope.go:117] "RemoveContainer" containerID="1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9" Jul 23 19:23:26.967301 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.967275 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" event={"ID":"a0d629ce-0a9c-48c6-accd-87ab1e5e7685","Type":"ContainerStarted","Data":"1da987bf9eff7d0529e9fc74190eb3a57d39b33b8a06c133cad68757cb1e65bf"} Jul 23 19:23:26.969236 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.969212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" event={"ID":"b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e","Type":"ContainerStarted","Data":"4720f5f9b6d1b6a1e7df87f22a6a12e91a68ea1a1023df73ab131b2a534b54f8"} Jul 23 19:23:26.969358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.969255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" event={"ID":"b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e","Type":"ContainerStarted","Data":"8367b14de68637bd24c33e89d879e5e8c80cb9b3c199b816d027a2162baffa2c"} Jul 23 19:23:26.969358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.969284 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:26.970503 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.970474 2574 generic.go:356] "Generic (PLEG): container finished" podID="04973124-1eb3-437d-b40c-e82f0e2e39e9" containerID="629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb" exitCode=0 Jul 23 19:23:26.970596 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.970511 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:26.975206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.975188 2574 scope.go:117] "RemoveContainer" containerID="1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9" Jul 23 19:23:26.975485 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:26.975469 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9\": container with ID starting with 1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9 not found: ID does not exist" containerID="1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9" Jul 23 19:23:26.975573 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.975492 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9"} err="failed to get container status \"1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9\": rpc error: code = NotFound desc = could not find container \"1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9\": container with ID starting with 1651a89dc6d036b123dfe38aff566cc4572f9af84d1c44da3a98ef449ca8f4a9 not found: ID does not exist" Jul 23 19:23:26.975573 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.975510 2574 scope.go:117] "RemoveContainer" containerID="629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb" Jul 23 19:23:26.983474 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.983453 2574 scope.go:117] "RemoveContainer" containerID="629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb" Jul 23 19:23:26.983761 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:26.983742 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb\": container with ID starting with 629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb not found: ID does not exist" containerID="629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb" Jul 23 19:23:26.983818 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:26.983770 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb"} err="failed to get container status \"629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb\": rpc error: code = NotFound desc = could not find container \"629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb\": container with ID starting with 629bb0a3cd9302e9fa0253861f4531c5190e86a3c3b8b7a155ef0f7654ba14bb not found: ID does not exist" Jul 23 19:23:27.005462 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:27.005403 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" podStartSLOduration=14.005383361 podStartE2EDuration="14.005383361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:27.004212243 +0000 UTC m=+454.576832785" watchObservedRunningTime="2026-07-23 19:23:27.005383361 +0000 UTC m=+454.578003954" Jul 23 19:23:27.048902 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:27.048857 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2chvn\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="b6f16b3d-8caf-465b-9f58-e34a894c3583" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2chvn" Jul 23 19:23:27.050838 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:27.050812 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kf9xq\" is forbidden: User \"system:node:ip-10-0-133-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-134.ec2.internal' and this object" podUID="04973124-1eb3-437d-b40c-e82f0e2e39e9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kf9xq" Jul 23 19:23:27.095235 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:27.095182 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" podStartSLOduration=14.095163592 podStartE2EDuration="14.095163592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:27.092682763 +0000 UTC m=+454.665303303" watchObservedRunningTime="2026-07-23 19:23:27.095163592 +0000 UTC m=+454.667784131" Jul 23 19:23:27.095430 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:27.095410 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tgd69" podStartSLOduration=1.469917434 podStartE2EDuration="24.095404516s" totalImagesPullingTime="22.625487082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:27.046009641 +0000 UTC m=+454.618630180" watchObservedRunningTime="2026-07-23 19:23:27.095404516 +0000 UTC m=+454.668025056" Jul 23 19:23:31.325819 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.325781 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8"] Jul 23 19:23:31.326280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.325977 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" podUID="4e8f1633-d9a6-4edc-8336-85457abb39e8" containerName="manager" containerID="cri-o://d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e" gracePeriod=10 Jul 23 19:23:31.327881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.327857 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:31.581152 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.581078 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:31.714024 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.713983 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwfkj\" (UniqueName: \"kubernetes.io/projected/4e8f1633-d9a6-4edc-8336-85457abb39e8-kube-api-access-pwfkj\") pod \"4e8f1633-d9a6-4edc-8336-85457abb39e8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " Jul 23 19:23:31.714226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.714068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e8f1633-d9a6-4edc-8336-85457abb39e8-extensions-socket-volume\") pod \"4e8f1633-d9a6-4edc-8336-85457abb39e8\" (UID: \"4e8f1633-d9a6-4edc-8336-85457abb39e8\") " Jul 23 19:23:31.714448 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.714422 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8f1633-d9a6-4edc-8336-85457abb39e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4e8f1633-d9a6-4edc-8336-85457abb39e8" (UID: "4e8f1633-d9a6-4edc-8336-85457abb39e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:23:31.716219 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.716195 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8f1633-d9a6-4edc-8336-85457abb39e8-kube-api-access-pwfkj" (OuterVolumeSpecName: "kube-api-access-pwfkj") pod "4e8f1633-d9a6-4edc-8336-85457abb39e8" (UID: "4e8f1633-d9a6-4edc-8336-85457abb39e8"). InnerVolumeSpecName "kube-api-access-pwfkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:23:31.815038 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.815003 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfkj\" (UniqueName: \"kubernetes.io/projected/4e8f1633-d9a6-4edc-8336-85457abb39e8-kube-api-access-pwfkj\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:23:31.815038 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.815031 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e8f1633-d9a6-4edc-8336-85457abb39e8-extensions-socket-volume\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:23:31.998013 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.997976 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e8f1633-d9a6-4edc-8336-85457abb39e8" containerID="d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e" exitCode=0 Jul 23 19:23:31.998202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.998020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" event={"ID":"4e8f1633-d9a6-4edc-8336-85457abb39e8","Type":"ContainerDied","Data":"d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e"} Jul 23 19:23:31.998202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.998063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" event={"ID":"4e8f1633-d9a6-4edc-8336-85457abb39e8","Type":"ContainerDied","Data":"9dd618b3fd068baec1e16d283f12b4a29b90dd6dd9b662a6fd023176e9315d00"} Jul 23 19:23:31.998202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.998087 2574 scope.go:117] "RemoveContainer" containerID="d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e" Jul 23 19:23:31.998202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:31.998025 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8" Jul 23 19:23:32.007107 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:32.007084 2574 scope.go:117] "RemoveContainer" containerID="d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e" Jul 23 19:23:32.007404 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:32.007385 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e\": container with ID starting with d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e not found: ID does not exist" containerID="d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e" Jul 23 19:23:32.007458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:32.007413 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e"} err="failed to get container status \"d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e\": rpc error: code = NotFound desc = could not find container \"d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e\": container with ID starting with d5d2486719f736ceb64beb89ad48d011d0302bc06c370e496c83cd7ce6ea9b7e not found: ID does not exist" Jul 23 19:23:32.023286 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:32.023239 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8"] Jul 23 19:23:32.025854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:32.025828 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2src8"] Jul 23 19:23:32.847227 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:32.847096 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e8f1633-d9a6-4edc-8336-85457abb39e8" path="/var/lib/kubelet/pods/4e8f1633-d9a6-4edc-8336-85457abb39e8/volumes" Jul 23 19:23:37.981300 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:37.981271 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8zwxj" Jul 23 19:23:47.761330 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.761293 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn"] Jul 23 19:23:47.761714 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.761657 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e8f1633-d9a6-4edc-8336-85457abb39e8" containerName="manager" Jul 23 19:23:47.761714 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.761668 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e8f1633-d9a6-4edc-8336-85457abb39e8" containerName="manager" Jul 23 19:23:47.761805 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.761744 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e8f1633-d9a6-4edc-8336-85457abb39e8" containerName="manager" Jul 23 19:23:47.764410 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.764392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.767384 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.767362 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96h6b\"" Jul 23 19:23:47.782356 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.782323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn"] Jul 23 19:23:47.952825 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.952794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0271634-a745-49c3-a960-5e182d11e8de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.952989 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.952841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0271634-a745-49c3-a960-5e182d11e8de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.952989 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.952907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.952989 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.952960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.953093 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.952992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.953093 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.953057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwcm\" (UniqueName: \"kubernetes.io/projected/f0271634-a745-49c3-a960-5e182d11e8de-kube-api-access-cvwcm\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.953159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.953115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.953159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.953142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0271634-a745-49c3-a960-5e182d11e8de-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:47.953221 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:47.953183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054602 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwcm\" (UniqueName: \"kubernetes.io/projected/f0271634-a745-49c3-a960-5e182d11e8de-kube-api-access-cvwcm\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0271634-a745-49c3-a960-5e182d11e8de-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0271634-a745-49c3-a960-5e182d11e8de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.054934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0271634-a745-49c3-a960-5e182d11e8de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.055232 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.054980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.055232 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.055112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.055232 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.055207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.055375 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.055269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.055535 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.055510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0271634-a745-49c3-a960-5e182d11e8de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.056850 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.056828 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0271634-a745-49c3-a960-5e182d11e8de-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.057210 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.057191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0271634-a745-49c3-a960-5e182d11e8de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.062163 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.062143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0271634-a745-49c3-a960-5e182d11e8de-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.062451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.062429 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwcm\" (UniqueName: \"kubernetes.io/projected/f0271634-a745-49c3-a960-5e182d11e8de-kube-api-access-cvwcm\") pod \"maas-default-gateway-openshift-default-56654dcf7-wwjfn\" (UID: \"f0271634-a745-49c3-a960-5e182d11e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.075053 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.075031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:48.208372 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.208349 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn"] Jul 23 19:23:48.210515 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:23:48.210488 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0271634_a745_49c3_a960_5e182d11e8de.slice/crio-0eddd31481f2cf26d1ccb28ca0e57a18089e3d3d3df19c1627500fe506c4daee WatchSource:0}: Error finding container 0eddd31481f2cf26d1ccb28ca0e57a18089e3d3d3df19c1627500fe506c4daee: Status 404 returned error can't find the container with id 0eddd31481f2cf26d1ccb28ca0e57a18089e3d3d3df19c1627500fe506c4daee Jul 23 19:23:48.212779 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.212747 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:23:48.212856 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.212811 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:23:48.212856 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:48.212840 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:23:49.080434 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:49.080399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" event={"ID":"f0271634-a745-49c3-a960-5e182d11e8de","Type":"ContainerStarted","Data":"202e83e9d63b2a54b89e135958b4bdf3c0d6d431bb6135ee6ffcd0aa6276df13"} Jul 23 19:23:49.080897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:49.080458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" event={"ID":"f0271634-a745-49c3-a960-5e182d11e8de","Type":"ContainerStarted","Data":"0eddd31481f2cf26d1ccb28ca0e57a18089e3d3d3df19c1627500fe506c4daee"} Jul 23 19:23:49.101509 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:49.101458 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" podStartSLOduration=2.101442912 podStartE2EDuration="2.101442912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:49.101082618 +0000 UTC m=+476.673703168" watchObservedRunningTime="2026-07-23 19:23:49.101442912 +0000 UTC m=+476.674063449" Jul 23 19:23:50.076028 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.075994 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:50.080925 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.080901 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:50.085700 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.085676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:50.086695 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.086676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:23:50.107407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.107374 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:23:50.110997 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.110975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.113133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.113114 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 19:23:50.120756 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.120716 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:23:50.173723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.173687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c560f64a-4975-4593-8642-146d83652354-config-file\") pod \"limitador-limitador-64c94fd64f-59k9d\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.173920 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.173786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smq5c\" (UniqueName: \"kubernetes.io/projected/c560f64a-4975-4593-8642-146d83652354-kube-api-access-smq5c\") pod \"limitador-limitador-64c94fd64f-59k9d\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.219428 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.218854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:23:50.275164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.275115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c560f64a-4975-4593-8642-146d83652354-config-file\") pod \"limitador-limitador-64c94fd64f-59k9d\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.275344 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.275176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smq5c\" (UniqueName: \"kubernetes.io/projected/c560f64a-4975-4593-8642-146d83652354-kube-api-access-smq5c\") pod \"limitador-limitador-64c94fd64f-59k9d\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.275780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.275726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c560f64a-4975-4593-8642-146d83652354-config-file\") pod \"limitador-limitador-64c94fd64f-59k9d\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.283854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.283827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smq5c\" (UniqueName: \"kubernetes.io/projected/c560f64a-4975-4593-8642-146d83652354-kube-api-access-smq5c\") pod \"limitador-limitador-64c94fd64f-59k9d\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.423088 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.423053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:50.583439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:50.583279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:23:50.586191 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:23:50.586162 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc560f64a_4975_4593_8642_146d83652354.slice/crio-24cd05ad589f3b65dfeb7039b7514d29a9d4d1e00de2139b673e6fba5988f810 WatchSource:0}: Error finding container 24cd05ad589f3b65dfeb7039b7514d29a9d4d1e00de2139b673e6fba5988f810: Status 404 returned error can't find the container with id 24cd05ad589f3b65dfeb7039b7514d29a9d4d1e00de2139b673e6fba5988f810 Jul 23 19:23:51.091975 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.091925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" event={"ID":"c560f64a-4975-4593-8642-146d83652354","Type":"ContainerStarted","Data":"24cd05ad589f3b65dfeb7039b7514d29a9d4d1e00de2139b673e6fba5988f810"} Jul 23 19:23:51.097778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.097752 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qshdm"] Jul 23 19:23:51.102555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.102534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:51.106073 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.106053 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgfl8\"" Jul 23 19:23:51.109497 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.109473 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qshdm"] Jul 23 19:23:51.180325 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.180289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmk7\" (UniqueName: \"kubernetes.io/projected/4ce0622c-0a59-47be-a70c-30774ca30110-kube-api-access-5lmk7\") pod \"authorino-58cd955675-qshdm\" (UID: \"4ce0622c-0a59-47be-a70c-30774ca30110\") " pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:51.281077 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.281042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmk7\" (UniqueName: \"kubernetes.io/projected/4ce0622c-0a59-47be-a70c-30774ca30110-kube-api-access-5lmk7\") pod \"authorino-58cd955675-qshdm\" (UID: \"4ce0622c-0a59-47be-a70c-30774ca30110\") " pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:51.291493 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.291333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmk7\" (UniqueName: \"kubernetes.io/projected/4ce0622c-0a59-47be-a70c-30774ca30110-kube-api-access-5lmk7\") pod \"authorino-58cd955675-qshdm\" (UID: \"4ce0622c-0a59-47be-a70c-30774ca30110\") " pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:51.413324 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.413292 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:51.602184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:51.602143 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qshdm"] Jul 23 19:23:51.605787 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:23:51.605718 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ce0622c_0a59_47be_a70c_30774ca30110.slice/crio-bb099728c8f77ae3ca0e9a569a90987b7b7f3c4525d0a155b777beefe8da8db8 WatchSource:0}: Error finding container bb099728c8f77ae3ca0e9a569a90987b7b7f3c4525d0a155b777beefe8da8db8: Status 404 returned error can't find the container with id bb099728c8f77ae3ca0e9a569a90987b7b7f3c4525d0a155b777beefe8da8db8 Jul 23 19:23:52.098819 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:52.098783 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qshdm" event={"ID":"4ce0622c-0a59-47be-a70c-30774ca30110","Type":"ContainerStarted","Data":"bb099728c8f77ae3ca0e9a569a90987b7b7f3c4525d0a155b777beefe8da8db8"} Jul 23 19:23:54.111776 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:54.111717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" event={"ID":"c560f64a-4975-4593-8642-146d83652354","Type":"ContainerStarted","Data":"fc07ba1cecc63daeac36bf09ddd99085212620c42058a74e3a50d4ef473afdc3"} Jul 23 19:23:54.133422 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:54.133362 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" podStartSLOduration=1.174693687 podStartE2EDuration="4.133344279s" totalImagesPullingTime="2.958650592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:54.131178474 +0000 UTC m=+481.703799014" watchObservedRunningTime="2026-07-23 19:23:54.133344279 +0000 UTC m=+481.705964869" Jul 23 19:23:55.117336 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:55.117302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:23:55.487889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:55.487853 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qshdm"] Jul 23 19:23:56.122929 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:56.122892 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qshdm" event={"ID":"4ce0622c-0a59-47be-a70c-30774ca30110","Type":"ContainerStarted","Data":"64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146"} Jul 23 19:23:56.147120 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:56.147062 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qshdm" podStartSLOduration=1.389201206 podStartE2EDuration="5.147043196s" totalImagesPullingTime="3.75784199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:23:56.145551317 +0000 UTC m=+483.718171858" watchObservedRunningTime="2026-07-23 19:23:56.147043196 +0000 UTC m=+483.719663735" Jul 23 19:23:57.127267 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:57.127229 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qshdm" podUID="4ce0622c-0a59-47be-a70c-30774ca30110" containerName="authorino" containerID="cri-o://64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146" gracePeriod=30 Jul 23 19:23:57.386304 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:57.386235 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:57.435845 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:57.435810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lmk7\" (UniqueName: \"kubernetes.io/projected/4ce0622c-0a59-47be-a70c-30774ca30110-kube-api-access-5lmk7\") pod \"4ce0622c-0a59-47be-a70c-30774ca30110\" (UID: \"4ce0622c-0a59-47be-a70c-30774ca30110\") " Jul 23 19:23:57.437868 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:57.437843 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce0622c-0a59-47be-a70c-30774ca30110-kube-api-access-5lmk7" (OuterVolumeSpecName: "kube-api-access-5lmk7") pod "4ce0622c-0a59-47be-a70c-30774ca30110" (UID: "4ce0622c-0a59-47be-a70c-30774ca30110"). InnerVolumeSpecName "kube-api-access-5lmk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:23:57.536436 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:57.536401 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmk7\" (UniqueName: \"kubernetes.io/projected/4ce0622c-0a59-47be-a70c-30774ca30110-kube-api-access-5lmk7\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:23:58.133001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.132968 2574 generic.go:356] "Generic (PLEG): container finished" podID="4ce0622c-0a59-47be-a70c-30774ca30110" containerID="64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146" exitCode=0 Jul 23 19:23:58.133451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.133012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qshdm" event={"ID":"4ce0622c-0a59-47be-a70c-30774ca30110","Type":"ContainerDied","Data":"64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146"} Jul 23 19:23:58.133451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.133016 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qshdm" Jul 23 19:23:58.133451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.133044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qshdm" event={"ID":"4ce0622c-0a59-47be-a70c-30774ca30110","Type":"ContainerDied","Data":"bb099728c8f77ae3ca0e9a569a90987b7b7f3c4525d0a155b777beefe8da8db8"} Jul 23 19:23:58.133451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.133062 2574 scope.go:117] "RemoveContainer" containerID="64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146" Jul 23 19:23:58.141689 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.141663 2574 scope.go:117] "RemoveContainer" containerID="64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146" Jul 23 19:23:58.141993 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:23:58.141971 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146\": container with ID starting with 64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146 not found: ID does not exist" containerID="64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146" Jul 23 19:23:58.142046 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.142001 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146"} err="failed to get container status \"64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146\": rpc error: code = NotFound desc = could not find container \"64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146\": container with ID starting with 64f71781656b534cb47f904fbf55de63321136601703920157edc016d3385146 not found: ID does not exist" Jul 23 19:23:58.160992 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.160961 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qshdm"] Jul 23 19:23:58.163784 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.163756 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qshdm"] Jul 23 19:23:58.847031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:23:58.847000 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce0622c-0a59-47be-a70c-30774ca30110" path="/var/lib/kubelet/pods/4ce0622c-0a59-47be-a70c-30774ca30110/volumes" Jul 23 19:24:05.432549 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:05.432515 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:24:05.432957 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:05.432707 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" podUID="c560f64a-4975-4593-8642-146d83652354" containerName="limitador" containerID="cri-o://fc07ba1cecc63daeac36bf09ddd99085212620c42058a74e3a50d4ef473afdc3" gracePeriod=30 Jul 23 19:24:05.433390 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:05.433373 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:24:06.179352 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.179321 2574 generic.go:356] "Generic (PLEG): container finished" podID="c560f64a-4975-4593-8642-146d83652354" containerID="fc07ba1cecc63daeac36bf09ddd99085212620c42058a74e3a50d4ef473afdc3" exitCode=0 Jul 23 19:24:06.179525 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.179385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" event={"ID":"c560f64a-4975-4593-8642-146d83652354","Type":"ContainerDied","Data":"fc07ba1cecc63daeac36bf09ddd99085212620c42058a74e3a50d4ef473afdc3"} Jul 23 19:24:06.380649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.380624 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:24:06.410054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.410026 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c560f64a-4975-4593-8642-146d83652354-config-file\") pod \"c560f64a-4975-4593-8642-146d83652354\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " Jul 23 19:24:06.410226 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.410076 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smq5c\" (UniqueName: \"kubernetes.io/projected/c560f64a-4975-4593-8642-146d83652354-kube-api-access-smq5c\") pod \"c560f64a-4975-4593-8642-146d83652354\" (UID: \"c560f64a-4975-4593-8642-146d83652354\") " Jul 23 19:24:06.410371 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.410346 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c560f64a-4975-4593-8642-146d83652354-config-file" (OuterVolumeSpecName: "config-file") pod "c560f64a-4975-4593-8642-146d83652354" (UID: "c560f64a-4975-4593-8642-146d83652354"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:24:06.412205 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.412179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c560f64a-4975-4593-8642-146d83652354-kube-api-access-smq5c" (OuterVolumeSpecName: "kube-api-access-smq5c") pod "c560f64a-4975-4593-8642-146d83652354" (UID: "c560f64a-4975-4593-8642-146d83652354"). InnerVolumeSpecName "kube-api-access-smq5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:24:06.511032 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.510948 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c560f64a-4975-4593-8642-146d83652354-config-file\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:24:06.511032 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:06.510980 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smq5c\" (UniqueName: \"kubernetes.io/projected/c560f64a-4975-4593-8642-146d83652354-kube-api-access-smq5c\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:24:07.185783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:07.185747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" event={"ID":"c560f64a-4975-4593-8642-146d83652354","Type":"ContainerDied","Data":"24cd05ad589f3b65dfeb7039b7514d29a9d4d1e00de2139b673e6fba5988f810"} Jul 23 19:24:07.185783 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:07.185777 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-59k9d" Jul 23 19:24:07.186000 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:07.185811 2574 scope.go:117] "RemoveContainer" containerID="fc07ba1cecc63daeac36bf09ddd99085212620c42058a74e3a50d4ef473afdc3" Jul 23 19:24:07.203739 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:07.203697 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:24:07.208077 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:07.208052 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-59k9d"] Jul 23 19:24:08.846914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:08.846882 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c560f64a-4975-4593-8642-146d83652354" path="/var/lib/kubelet/pods/c560f64a-4975-4593-8642-146d83652354/volumes" Jul 23 19:24:11.836158 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836126 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq"] Jul 23 19:24:11.836575 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836556 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce0622c-0a59-47be-a70c-30774ca30110" containerName="authorino" Jul 23 19:24:11.836575 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836577 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce0622c-0a59-47be-a70c-30774ca30110" containerName="authorino" Jul 23 19:24:11.836695 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836585 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c560f64a-4975-4593-8642-146d83652354" containerName="limitador" Jul 23 19:24:11.836695 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836593 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c560f64a-4975-4593-8642-146d83652354" containerName="limitador" Jul 23 19:24:11.836808 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836694 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c560f64a-4975-4593-8642-146d83652354" containerName="limitador" Jul 23 19:24:11.836808 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.836709 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce0622c-0a59-47be-a70c-30774ca30110" containerName="authorino" Jul 23 19:24:11.841367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.841347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:11.843720 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.843699 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 19:24:11.844899 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.844875 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 19:24:11.845021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.844968 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 19:24:11.845130 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.845116 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kkppv\"" Jul 23 19:24:11.848152 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.848132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq"] Jul 23 19:24:11.960570 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.960532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5209404-da52-42e3-845f-e1767d3ca080-data\") pod \"postgres-69fb4594cf-k8tsq\" (UID: \"e5209404-da52-42e3-845f-e1767d3ca080\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:11.960716 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:11.960579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt4wc\" (UniqueName: \"kubernetes.io/projected/e5209404-da52-42e3-845f-e1767d3ca080-kube-api-access-xt4wc\") pod \"postgres-69fb4594cf-k8tsq\" (UID: \"e5209404-da52-42e3-845f-e1767d3ca080\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:12.061274 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:12.061232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5209404-da52-42e3-845f-e1767d3ca080-data\") pod \"postgres-69fb4594cf-k8tsq\" (UID: \"e5209404-da52-42e3-845f-e1767d3ca080\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:12.061422 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:12.061286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt4wc\" (UniqueName: \"kubernetes.io/projected/e5209404-da52-42e3-845f-e1767d3ca080-kube-api-access-xt4wc\") pod \"postgres-69fb4594cf-k8tsq\" (UID: \"e5209404-da52-42e3-845f-e1767d3ca080\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:12.061615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:12.061598 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5209404-da52-42e3-845f-e1767d3ca080-data\") pod \"postgres-69fb4594cf-k8tsq\" (UID: \"e5209404-da52-42e3-845f-e1767d3ca080\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:12.070889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:12.070867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt4wc\" (UniqueName: \"kubernetes.io/projected/e5209404-da52-42e3-845f-e1767d3ca080-kube-api-access-xt4wc\") pod \"postgres-69fb4594cf-k8tsq\" (UID: \"e5209404-da52-42e3-845f-e1767d3ca080\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:12.153248 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:12.153150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:12.297847 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:12.297822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq"] Jul 23 19:24:12.300065 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:24:12.300039 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5209404_da52_42e3_845f_e1767d3ca080.slice/crio-58b0c4d131e4593d19fa0a97c3ed9f9da9a7bbf2f8d5452af4b4826a6c476c06 WatchSource:0}: Error finding container 58b0c4d131e4593d19fa0a97c3ed9f9da9a7bbf2f8d5452af4b4826a6c476c06: Status 404 returned error can't find the container with id 58b0c4d131e4593d19fa0a97c3ed9f9da9a7bbf2f8d5452af4b4826a6c476c06 Jul 23 19:24:13.217462 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:13.217399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" event={"ID":"e5209404-da52-42e3-845f-e1767d3ca080","Type":"ContainerStarted","Data":"58b0c4d131e4593d19fa0a97c3ed9f9da9a7bbf2f8d5452af4b4826a6c476c06"} Jul 23 19:24:18.247607 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:18.247561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" event={"ID":"e5209404-da52-42e3-845f-e1767d3ca080","Type":"ContainerStarted","Data":"f371d7569d643bde1f813af41f35decf8a479b3ecdd22f6c9d3d37fed963dfe9"} Jul 23 19:24:18.248016 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:18.247722 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:18.263188 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:18.263141 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" podStartSLOduration=2.281815548 podStartE2EDuration="7.26312658s" totalImagesPullingTime="4.981311032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:24:18.261875838 +0000 UTC m=+505.834496379" watchObservedRunningTime="2026-07-23 19:24:18.26312658 +0000 UTC m=+505.835747120" Jul 23 19:24:24.283587 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:24.283555 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k8tsq" Jul 23 19:24:24.823083 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:24.823045 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-24hsj"] Jul 23 19:24:24.858831 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:24.858791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:24.861563 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:24.861541 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgfl8\"" Jul 23 19:24:24.865482 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:24.865457 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-24hsj"] Jul 23 19:24:24.984308 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:24.984273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hgt\" (UniqueName: \"kubernetes.io/projected/aeaebbea-d792-4807-a95d-cbd0a78defdb-kube-api-access-s7hgt\") pod \"authorino-5d4b67c446-24hsj\" (UID: \"aeaebbea-d792-4807-a95d-cbd0a78defdb\") " pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:25.072208 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.072171 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-24hsj"] Jul 23 19:24:25.072466 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:24:25.072446 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s7hgt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-24hsj" podUID="aeaebbea-d792-4807-a95d-cbd0a78defdb" Jul 23 19:24:25.085809 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.085715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hgt\" (UniqueName: \"kubernetes.io/projected/aeaebbea-d792-4807-a95d-cbd0a78defdb-kube-api-access-s7hgt\") pod \"authorino-5d4b67c446-24hsj\" (UID: \"aeaebbea-d792-4807-a95d-cbd0a78defdb\") " pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:25.101973 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.101937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hgt\" (UniqueName: \"kubernetes.io/projected/aeaebbea-d792-4807-a95d-cbd0a78defdb-kube-api-access-s7hgt\") pod \"authorino-5d4b67c446-24hsj\" (UID: \"aeaebbea-d792-4807-a95d-cbd0a78defdb\") " pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:25.103435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.103414 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6498dbb5dd-z76x2"] Jul 23 19:24:25.141912 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.141872 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6498dbb5dd-z76x2"] Jul 23 19:24:25.141912 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.141899 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:25.275680 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.275642 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6498dbb5dd-z76x2"] Jul 23 19:24:25.275952 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:24:25.275931 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fz4jt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6498dbb5dd-z76x2" podUID="790a4a13-413f-4652-b30e-24435d3f05da" Jul 23 19:24:25.282007 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.281977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:25.282155 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.281982 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:25.287029 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.286993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4jt\" (UniqueName: \"kubernetes.io/projected/790a4a13-413f-4652-b30e-24435d3f05da-kube-api-access-fz4jt\") pod \"authorino-6498dbb5dd-z76x2\" (UID: \"790a4a13-413f-4652-b30e-24435d3f05da\") " pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:25.305406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.305357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:25.310081 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.310057 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c885856f6-6rr2p"] Jul 23 19:24:25.321421 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.321398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:25.335130 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.335091 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c885856f6-6rr2p"] Jul 23 19:24:25.335405 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.335383 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.338334 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.338267 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 19:24:25.387603 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.387570 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7hgt\" (UniqueName: \"kubernetes.io/projected/aeaebbea-d792-4807-a95d-cbd0a78defdb-kube-api-access-s7hgt\") pod \"aeaebbea-d792-4807-a95d-cbd0a78defdb\" (UID: \"aeaebbea-d792-4807-a95d-cbd0a78defdb\") " Jul 23 19:24:25.387802 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.387785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4jt\" (UniqueName: \"kubernetes.io/projected/790a4a13-413f-4652-b30e-24435d3f05da-kube-api-access-fz4jt\") pod \"authorino-6498dbb5dd-z76x2\" (UID: \"790a4a13-413f-4652-b30e-24435d3f05da\") " pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:25.389775 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.389751 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaebbea-d792-4807-a95d-cbd0a78defdb-kube-api-access-s7hgt" (OuterVolumeSpecName: "kube-api-access-s7hgt") pod "aeaebbea-d792-4807-a95d-cbd0a78defdb" (UID: "aeaebbea-d792-4807-a95d-cbd0a78defdb"). InnerVolumeSpecName "kube-api-access-s7hgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:24:25.398942 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.398916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4jt\" (UniqueName: \"kubernetes.io/projected/790a4a13-413f-4652-b30e-24435d3f05da-kube-api-access-fz4jt\") pod \"authorino-6498dbb5dd-z76x2\" (UID: \"790a4a13-413f-4652-b30e-24435d3f05da\") " pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:25.488841 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.488808 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz4jt\" (UniqueName: \"kubernetes.io/projected/790a4a13-413f-4652-b30e-24435d3f05da-kube-api-access-fz4jt\") pod \"790a4a13-413f-4652-b30e-24435d3f05da\" (UID: \"790a4a13-413f-4652-b30e-24435d3f05da\") " Jul 23 19:24:25.489016 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.488995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-tls-cert\") pod \"authorino-c885856f6-6rr2p\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.489085 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.489070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgtf\" (UniqueName: \"kubernetes.io/projected/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-kube-api-access-wdgtf\") pod \"authorino-c885856f6-6rr2p\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.489164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.489150 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hgt\" (UniqueName: \"kubernetes.io/projected/aeaebbea-d792-4807-a95d-cbd0a78defdb-kube-api-access-s7hgt\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:24:25.490984 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.490959 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790a4a13-413f-4652-b30e-24435d3f05da-kube-api-access-fz4jt" (OuterVolumeSpecName: "kube-api-access-fz4jt") pod "790a4a13-413f-4652-b30e-24435d3f05da" (UID: "790a4a13-413f-4652-b30e-24435d3f05da"). InnerVolumeSpecName "kube-api-access-fz4jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:24:25.590350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.590257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-tls-cert\") pod \"authorino-c885856f6-6rr2p\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.590350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.590323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgtf\" (UniqueName: \"kubernetes.io/projected/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-kube-api-access-wdgtf\") pod \"authorino-c885856f6-6rr2p\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.590540 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.590375 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4jt\" (UniqueName: \"kubernetes.io/projected/790a4a13-413f-4652-b30e-24435d3f05da-kube-api-access-fz4jt\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:24:25.592710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.592679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-tls-cert\") pod \"authorino-c885856f6-6rr2p\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.598619 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.598588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgtf\" (UniqueName: \"kubernetes.io/projected/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-kube-api-access-wdgtf\") pod \"authorino-c885856f6-6rr2p\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.645433 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.645402 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:24:25.808861 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:25.808834 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c885856f6-6rr2p"] Jul 23 19:24:25.811014 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:24:25.810990 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd96846a4_0aa9_4136_bdce_72c52e9a0d4c.slice/crio-02a969b3f6808f51559dc14a07a1c91b56092298d35e302774546e5396021a1a WatchSource:0}: Error finding container 02a969b3f6808f51559dc14a07a1c91b56092298d35e302774546e5396021a1a: Status 404 returned error can't find the container with id 02a969b3f6808f51559dc14a07a1c91b56092298d35e302774546e5396021a1a Jul 23 19:24:26.287963 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.287928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885856f6-6rr2p" event={"ID":"d96846a4-0aa9-4136-bdce-72c52e9a0d4c","Type":"ContainerStarted","Data":"02a969b3f6808f51559dc14a07a1c91b56092298d35e302774546e5396021a1a"} Jul 23 19:24:26.288376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.287984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498dbb5dd-z76x2" Jul 23 19:24:26.288376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.287987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-24hsj" Jul 23 19:24:26.324652 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.324618 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-24hsj"] Jul 23 19:24:26.328421 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.328395 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-24hsj"] Jul 23 19:24:26.349391 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.349361 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6498dbb5dd-z76x2"] Jul 23 19:24:26.355289 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.355258 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6498dbb5dd-z76x2"] Jul 23 19:24:26.847211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.847179 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790a4a13-413f-4652-b30e-24435d3f05da" path="/var/lib/kubelet/pods/790a4a13-413f-4652-b30e-24435d3f05da/volumes" Jul 23 19:24:26.847439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:26.847427 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeaebbea-d792-4807-a95d-cbd0a78defdb" path="/var/lib/kubelet/pods/aeaebbea-d792-4807-a95d-cbd0a78defdb/volumes" Jul 23 19:24:27.294627 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:27.294596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885856f6-6rr2p" event={"ID":"d96846a4-0aa9-4136-bdce-72c52e9a0d4c","Type":"ContainerStarted","Data":"c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032"} Jul 23 19:24:27.312379 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:27.312330 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c885856f6-6rr2p" podStartSLOduration=1.810567762 podStartE2EDuration="2.312313608s" totalImagesPullingTime="501.745846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:24:27.311833585 +0000 UTC m=+514.884454119" watchObservedRunningTime="2026-07-23 19:24:27.312313608 +0000 UTC m=+514.884934147" Jul 23 19:24:32.245648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.245612 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8fd995c4-pq7t7"] Jul 23 19:24:32.249919 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.249898 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.252846 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.252826 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mfldh\"" Jul 23 19:24:32.252969 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.252871 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 19:24:32.253113 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.253099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 19:24:32.265456 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.265436 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-pq7t7"] Jul 23 19:24:32.351532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.351495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.351705 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.351612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznz7\" (UniqueName: \"kubernetes.io/projected/56412035-bff1-456d-95a8-f9e71229f5fe-kube-api-access-zznz7\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.452232 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.452193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.452414 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.452281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznz7\" (UniqueName: \"kubernetes.io/projected/56412035-bff1-456d-95a8-f9e71229f5fe-kube-api-access-zznz7\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.452414 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:24:32.452349 2574 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 23 19:24:32.452506 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:24:32.452424 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert podName:56412035-bff1-456d-95a8-f9e71229f5fe nodeName:}" failed. No retries permitted until 2026-07-23 19:24:32.95240714 +0000 UTC m=+520.525027661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert") pod "maas-controller-c8fd995c4-pq7t7" (UID: "56412035-bff1-456d-95a8-f9e71229f5fe") : secret "maas-controller-webhook-cert" not found Jul 23 19:24:32.461883 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.461850 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznz7\" (UniqueName: \"kubernetes.io/projected/56412035-bff1-456d-95a8-f9e71229f5fe-kube-api-access-zznz7\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.956527 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.956491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:32.958854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:32.958832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert\") pod \"maas-controller-c8fd995c4-pq7t7\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:33.159437 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:33.159405 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:33.297529 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:33.297501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-pq7t7"] Jul 23 19:24:33.300171 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:24:33.300150 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56412035_bff1_456d_95a8_f9e71229f5fe.slice/crio-29a1392b51012e899f5d30e8e76b649721bfff29e45ca5b2e3c6e924c3e48a2d WatchSource:0}: Error finding container 29a1392b51012e899f5d30e8e76b649721bfff29e45ca5b2e3c6e924c3e48a2d: Status 404 returned error can't find the container with id 29a1392b51012e899f5d30e8e76b649721bfff29e45ca5b2e3c6e924c3e48a2d Jul 23 19:24:33.325210 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:33.325182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" event={"ID":"56412035-bff1-456d-95a8-f9e71229f5fe","Type":"ContainerStarted","Data":"29a1392b51012e899f5d30e8e76b649721bfff29e45ca5b2e3c6e924c3e48a2d"} Jul 23 19:24:36.343076 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:36.343039 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" event={"ID":"56412035-bff1-456d-95a8-f9e71229f5fe","Type":"ContainerStarted","Data":"71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120"} Jul 23 19:24:36.343583 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:36.343112 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:24:36.365989 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:36.365938 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" podStartSLOduration=2.077581626 podStartE2EDuration="4.365921356s" totalImagesPullingTime="2.28833973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:24:36.364145269 +0000 UTC m=+523.936765810" watchObservedRunningTime="2026-07-23 19:24:36.365921356 +0000 UTC m=+523.938541940" Jul 23 19:24:47.350097 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:24:47.350026 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:25:08.608363 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.608327 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7"] Jul 23 19:25:08.635349 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.635317 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7"] Jul 23 19:25:08.635508 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.635409 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.638647 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.638624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vk5pr\"" Jul 23 19:25:08.639356 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.639338 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 23 19:25:08.773133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.773101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621e05d1-6868-4488-a07e-87aa1f36b095-maas-api-tls\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.773133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.773138 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621e05d1-6868-4488-a07e-87aa1f36b095-openshift-service-ca\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.773359 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.773179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwcb\" (UniqueName: \"kubernetes.io/projected/621e05d1-6868-4488-a07e-87aa1f36b095-kube-api-access-fmwcb\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.873886 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.873807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwcb\" (UniqueName: \"kubernetes.io/projected/621e05d1-6868-4488-a07e-87aa1f36b095-kube-api-access-fmwcb\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.874059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.873927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621e05d1-6868-4488-a07e-87aa1f36b095-maas-api-tls\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.874059 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.873957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621e05d1-6868-4488-a07e-87aa1f36b095-openshift-service-ca\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.874659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.874636 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621e05d1-6868-4488-a07e-87aa1f36b095-openshift-service-ca\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.876488 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.876466 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621e05d1-6868-4488-a07e-87aa1f36b095-maas-api-tls\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.883715 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.883695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwcb\" (UniqueName: \"kubernetes.io/projected/621e05d1-6868-4488-a07e-87aa1f36b095-kube-api-access-fmwcb\") pod \"maas-api-79b7dbcf5c-69gl7\" (UID: \"621e05d1-6868-4488-a07e-87aa1f36b095\") " pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:08.946665 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:08.946630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:09.097817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.097790 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7"] Jul 23 19:25:09.099935 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:25:09.099905 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621e05d1_6868_4488_a07e_87aa1f36b095.slice/crio-3a8554829ad148a40b396e07fbbb8f38f0a62060eb9776c6d56a5829413a88e5 WatchSource:0}: Error finding container 3a8554829ad148a40b396e07fbbb8f38f0a62060eb9776c6d56a5829413a88e5: Status 404 returned error can't find the container with id 3a8554829ad148a40b396e07fbbb8f38f0a62060eb9776c6d56a5829413a88e5 Jul 23 19:25:09.236142 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.236108 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-npj46"] Jul 23 19:25:09.241006 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.240984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.243896 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.243869 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 19:25:09.244035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.243966 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bdv66\"" Jul 23 19:25:09.251933 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.251884 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-npj46"] Jul 23 19:25:09.378994 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.378965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f684b-7997-44d1-a1d6-9946459a79be-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-npj46\" (UID: \"6d3f684b-7997-44d1-a1d6-9946459a79be\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.379187 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.379027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtffc\" (UniqueName: \"kubernetes.io/projected/6d3f684b-7997-44d1-a1d6-9946459a79be-kube-api-access-qtffc\") pod \"payload-pre-processing-7677b774bb-npj46\" (UID: \"6d3f684b-7997-44d1-a1d6-9946459a79be\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.480212 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.480169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f684b-7997-44d1-a1d6-9946459a79be-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-npj46\" (UID: \"6d3f684b-7997-44d1-a1d6-9946459a79be\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.480389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.480260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtffc\" (UniqueName: \"kubernetes.io/projected/6d3f684b-7997-44d1-a1d6-9946459a79be-kube-api-access-qtffc\") pod \"payload-pre-processing-7677b774bb-npj46\" (UID: \"6d3f684b-7997-44d1-a1d6-9946459a79be\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.480884 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.480866 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f684b-7997-44d1-a1d6-9946459a79be-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-npj46\" (UID: \"6d3f684b-7997-44d1-a1d6-9946459a79be\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.494704 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.494648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtffc\" (UniqueName: \"kubernetes.io/projected/6d3f684b-7997-44d1-a1d6-9946459a79be-kube-api-access-qtffc\") pod \"payload-pre-processing-7677b774bb-npj46\" (UID: \"6d3f684b-7997-44d1-a1d6-9946459a79be\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.512311 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.512268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" event={"ID":"621e05d1-6868-4488-a07e-87aa1f36b095","Type":"ContainerStarted","Data":"3a8554829ad148a40b396e07fbbb8f38f0a62060eb9776c6d56a5829413a88e5"} Jul 23 19:25:09.554834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.554802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:09.698006 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:09.697973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-npj46"] Jul 23 19:25:09.703974 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:25:09.700648 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3f684b_7997_44d1_a1d6_9946459a79be.slice/crio-a767a6b5d71ceddc70b9bf7a4a88b69397795a06bc66593203a6538ff4cd2418 WatchSource:0}: Error finding container a767a6b5d71ceddc70b9bf7a4a88b69397795a06bc66593203a6538ff4cd2418: Status 404 returned error can't find the container with id a767a6b5d71ceddc70b9bf7a4a88b69397795a06bc66593203a6538ff4cd2418 Jul 23 19:25:10.522064 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:10.521973 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" event={"ID":"6d3f684b-7997-44d1-a1d6-9946459a79be","Type":"ContainerStarted","Data":"a767a6b5d71ceddc70b9bf7a4a88b69397795a06bc66593203a6538ff4cd2418"} Jul 23 19:25:11.552834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:11.552075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" event={"ID":"621e05d1-6868-4488-a07e-87aa1f36b095","Type":"ContainerStarted","Data":"6ba0ebb7e6291c3886fb92002b0dd26b2083d11af2a47637c5f340e1cf454f76"} Jul 23 19:25:11.552834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:11.552380 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:11.571823 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:11.571771 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" podStartSLOduration=1.487459134 podStartE2EDuration="3.571754185s" totalImagesPullingTime="2.084295051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:25:11.57000286 +0000 UTC m=+559.142623400" watchObservedRunningTime="2026-07-23 19:25:11.571754185 +0000 UTC m=+559.144374726" Jul 23 19:25:12.561065 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:12.561033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" event={"ID":"6d3f684b-7997-44d1-a1d6-9946459a79be","Type":"ContainerStarted","Data":"e001b7c125807c1c4b0eb7b8eb866a359a313be2bbeaef8542505add2bf0c85a"} Jul 23 19:25:12.561520 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:12.561088 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:12.576576 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:12.576511 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" podStartSLOduration=1.481392156 podStartE2EDuration="3.576494189s" totalImagesPullingTime="2.095102033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:25:12.575277701 +0000 UTC m=+560.147898252" watchObservedRunningTime="2026-07-23 19:25:12.576494189 +0000 UTC m=+560.149114731" Jul 23 19:25:17.566443 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.566410 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79b7dbcf5c-69gl7" Jul 23 19:25:17.587558 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.587528 2574 generic.go:356] "Generic (PLEG): container finished" podID="f0271634-a745-49c3-a960-5e182d11e8de" containerID="202e83e9d63b2a54b89e135958b4bdf3c0d6d431bb6135ee6ffcd0aa6276df13" exitCode=0 Jul 23 19:25:17.587725 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.587600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" event={"ID":"f0271634-a745-49c3-a960-5e182d11e8de","Type":"ContainerDied","Data":"202e83e9d63b2a54b89e135958b4bdf3c0d6d431bb6135ee6ffcd0aa6276df13"} Jul 23 19:25:17.588021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.588003 2574 scope.go:117] "RemoveContainer" containerID="202e83e9d63b2a54b89e135958b4bdf3c0d6d431bb6135ee6ffcd0aa6276df13" Jul 23 19:25:17.589042 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.589013 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:25:17.589102 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.589088 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:25:17.589139 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:17.589122 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:25:18.596119 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:18.596084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" event={"ID":"f0271634-a745-49c3-a960-5e182d11e8de","Type":"ContainerStarted","Data":"43f4b536399e5f0d70a572336c0daf968b9991bab1a4b668530cbd770e11fa3d"} Jul 23 19:25:19.075857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:19.075815 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:25:19.077035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:19.077009 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wwjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 19:25:19.077156 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:19.077061 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" podUID="f0271634-a745-49c3-a960-5e182d11e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 19:25:20.076160 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:20.076121 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wwjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 19:25:20.076586 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:20.076192 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" podUID="f0271634-a745-49c3-a960-5e182d11e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 19:25:20.086391 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:20.086361 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:25:21.075704 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:21.075659 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wwjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 19:25:21.075971 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:21.075786 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" podUID="f0271634-a745-49c3-a960-5e182d11e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 19:25:22.076031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:22.075994 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-wwjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 23 19:25:22.076433 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:22.076050 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" podUID="f0271634-a745-49c3-a960-5e182d11e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 23 19:25:23.080554 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:23.080523 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:25:23.081523 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:23.081503 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-wwjfn" Jul 23 19:25:23.568704 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:23.568671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-npj46" Jul 23 19:25:52.791678 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:52.791642 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:25:52.795153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:25:52.795130 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:27:28.836631 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:28.836589 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b8b59765-h7phg"] Jul 23 19:27:28.840230 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:28.840212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:28.854269 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:28.854240 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8b59765-h7phg"] Jul 23 19:27:28.921234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:28.921196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhhd\" (UniqueName: \"kubernetes.io/projected/21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff-kube-api-access-5qhhd\") pod \"authorino-5b8b59765-h7phg\" (UID: \"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff\") " pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:28.921428 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:28.921268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff-tls-cert\") pod \"authorino-5b8b59765-h7phg\" (UID: \"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff\") " pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:29.021958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.021916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhhd\" (UniqueName: \"kubernetes.io/projected/21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff-kube-api-access-5qhhd\") pod \"authorino-5b8b59765-h7phg\" (UID: \"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff\") " pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:29.022136 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.021994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff-tls-cert\") pod \"authorino-5b8b59765-h7phg\" (UID: \"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff\") " pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:29.024478 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.024456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff-tls-cert\") pod \"authorino-5b8b59765-h7phg\" (UID: \"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff\") " pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:29.030855 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.030824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhhd\" (UniqueName: \"kubernetes.io/projected/21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff-kube-api-access-5qhhd\") pod \"authorino-5b8b59765-h7phg\" (UID: \"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff\") " pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:29.150697 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.150616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8b59765-h7phg" Jul 23 19:27:29.286159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.286135 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8b59765-h7phg"] Jul 23 19:27:29.288567 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:27:29.288534 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21f2c2a2_d436_4a9a_8d4c_c5ba5c29baff.slice/crio-e8d8f83918ea959c9d53601cdd8bd609dd1bd4ec9a1102db02765872bd3af62c WatchSource:0}: Error finding container e8d8f83918ea959c9d53601cdd8bd609dd1bd4ec9a1102db02765872bd3af62c: Status 404 returned error can't find the container with id e8d8f83918ea959c9d53601cdd8bd609dd1bd4ec9a1102db02765872bd3af62c Jul 23 19:27:29.290240 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:29.290218 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:27:30.273427 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.273390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8b59765-h7phg" event={"ID":"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff","Type":"ContainerStarted","Data":"60794e8a406a66aa57a6f3f15d517e273a6776f198691b16d4db46dbeffd1156"} Jul 23 19:27:30.273851 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.273435 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8b59765-h7phg" event={"ID":"21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff","Type":"ContainerStarted","Data":"e8d8f83918ea959c9d53601cdd8bd609dd1bd4ec9a1102db02765872bd3af62c"} Jul 23 19:27:30.301925 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.301857 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b8b59765-h7phg" podStartSLOduration=1.788375073 podStartE2EDuration="2.30183635s" totalImagesPullingTime="513.461277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:27:30.299106609 +0000 UTC m=+697.871727150" watchObservedRunningTime="2026-07-23 19:27:30.30183635 +0000 UTC m=+697.874456891" Jul 23 19:27:30.333636 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.333597 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c885856f6-6rr2p"] Jul 23 19:27:30.335201 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.335168 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c885856f6-6rr2p" podUID="d96846a4-0aa9-4136-bdce-72c52e9a0d4c" containerName="authorino" containerID="cri-o://c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032" gracePeriod=30 Jul 23 19:27:30.611473 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.611447 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:27:30.637615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.637586 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdgtf\" (UniqueName: \"kubernetes.io/projected/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-kube-api-access-wdgtf\") pod \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " Jul 23 19:27:30.637815 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.637628 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-tls-cert\") pod \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\" (UID: \"d96846a4-0aa9-4136-bdce-72c52e9a0d4c\") " Jul 23 19:27:30.640063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.640024 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-kube-api-access-wdgtf" (OuterVolumeSpecName: "kube-api-access-wdgtf") pod "d96846a4-0aa9-4136-bdce-72c52e9a0d4c" (UID: "d96846a4-0aa9-4136-bdce-72c52e9a0d4c"). InnerVolumeSpecName "kube-api-access-wdgtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:27:30.649874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.649845 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d96846a4-0aa9-4136-bdce-72c52e9a0d4c" (UID: "d96846a4-0aa9-4136-bdce-72c52e9a0d4c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:27:30.738545 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.738510 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdgtf\" (UniqueName: \"kubernetes.io/projected/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-kube-api-access-wdgtf\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:27:30.738545 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:30.738545 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d96846a4-0aa9-4136-bdce-72c52e9a0d4c-tls-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:27:31.280746 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.280698 2574 generic.go:356] "Generic (PLEG): container finished" podID="d96846a4-0aa9-4136-bdce-72c52e9a0d4c" containerID="c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032" exitCode=0 Jul 23 19:27:31.281190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.280758 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885856f6-6rr2p" Jul 23 19:27:31.281190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.280768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885856f6-6rr2p" event={"ID":"d96846a4-0aa9-4136-bdce-72c52e9a0d4c","Type":"ContainerDied","Data":"c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032"} Jul 23 19:27:31.281190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.280814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885856f6-6rr2p" event={"ID":"d96846a4-0aa9-4136-bdce-72c52e9a0d4c","Type":"ContainerDied","Data":"02a969b3f6808f51559dc14a07a1c91b56092298d35e302774546e5396021a1a"} Jul 23 19:27:31.281190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.280834 2574 scope.go:117] "RemoveContainer" containerID="c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032" Jul 23 19:27:31.290167 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.290141 2574 scope.go:117] "RemoveContainer" containerID="c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032" Jul 23 19:27:31.290478 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:27:31.290453 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032\": container with ID starting with c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032 not found: ID does not exist" containerID="c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032" Jul 23 19:27:31.290532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.290488 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032"} err="failed to get container status \"c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032\": rpc error: code = NotFound desc = could not find container \"c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032\": container with ID starting with c0c88d0dc16ae2600bfe446aefeeec6001355752af6183585d129ba7dfbd2032 not found: ID does not exist" Jul 23 19:27:31.300271 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.300240 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c885856f6-6rr2p"] Jul 23 19:27:31.306540 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:31.306502 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c885856f6-6rr2p"] Jul 23 19:27:32.848442 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:27:32.848412 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96846a4-0aa9-4136-bdce-72c52e9a0d4c" path="/var/lib/kubelet/pods/d96846a4-0aa9-4136-bdce-72c52e9a0d4c/volumes" Jul 23 19:28:17.902632 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:17.902599 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-pq7t7"] Jul 23 19:28:17.903159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:17.902870 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" podUID="56412035-bff1-456d-95a8-f9e71229f5fe" containerName="manager" containerID="cri-o://71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120" gracePeriod=30 Jul 23 19:28:18.151090 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.151064 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:28:18.246617 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.246580 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zznz7\" (UniqueName: \"kubernetes.io/projected/56412035-bff1-456d-95a8-f9e71229f5fe-kube-api-access-zznz7\") pod \"56412035-bff1-456d-95a8-f9e71229f5fe\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " Jul 23 19:28:18.246852 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.246673 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert\") pod \"56412035-bff1-456d-95a8-f9e71229f5fe\" (UID: \"56412035-bff1-456d-95a8-f9e71229f5fe\") " Jul 23 19:28:18.248778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.248749 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "56412035-bff1-456d-95a8-f9e71229f5fe" (UID: "56412035-bff1-456d-95a8-f9e71229f5fe"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:28:18.248881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.248798 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56412035-bff1-456d-95a8-f9e71229f5fe-kube-api-access-zznz7" (OuterVolumeSpecName: "kube-api-access-zznz7") pod "56412035-bff1-456d-95a8-f9e71229f5fe" (UID: "56412035-bff1-456d-95a8-f9e71229f5fe"). InnerVolumeSpecName "kube-api-access-zznz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:28:18.347963 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.347922 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zznz7\" (UniqueName: \"kubernetes.io/projected/56412035-bff1-456d-95a8-f9e71229f5fe-kube-api-access-zznz7\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:28:18.347963 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.347956 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56412035-bff1-456d-95a8-f9e71229f5fe-webhook-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:28:18.510637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.510555 2574 generic.go:356] "Generic (PLEG): container finished" podID="56412035-bff1-456d-95a8-f9e71229f5fe" containerID="71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120" exitCode=0 Jul 23 19:28:18.510637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.510606 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" Jul 23 19:28:18.510827 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.510644 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" event={"ID":"56412035-bff1-456d-95a8-f9e71229f5fe","Type":"ContainerDied","Data":"71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120"} Jul 23 19:28:18.510827 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.510698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-pq7t7" event={"ID":"56412035-bff1-456d-95a8-f9e71229f5fe","Type":"ContainerDied","Data":"29a1392b51012e899f5d30e8e76b649721bfff29e45ca5b2e3c6e924c3e48a2d"} Jul 23 19:28:18.510827 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.510720 2574 scope.go:117] "RemoveContainer" containerID="71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120" Jul 23 19:28:18.521805 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.521784 2574 scope.go:117] "RemoveContainer" containerID="71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120" Jul 23 19:28:18.522101 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:28:18.522077 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120\": container with ID starting with 71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120 not found: ID does not exist" containerID="71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120" Jul 23 19:28:18.522180 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.522109 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120"} err="failed to get container status \"71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120\": rpc error: code = NotFound desc = could not find container \"71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120\": container with ID starting with 71763d53fc681eda4c0858e2901abc34f4bcce88a1976ec1615dfc7efb522120 not found: ID does not exist" Jul 23 19:28:18.544406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.544378 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-pq7t7"] Jul 23 19:28:18.553604 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.553574 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-pq7t7"] Jul 23 19:28:18.851221 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:28:18.851142 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56412035-bff1-456d-95a8-f9e71229f5fe" path="/var/lib/kubelet/pods/56412035-bff1-456d-95a8-f9e71229f5fe/volumes" Jul 23 19:29:19.354769 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.354668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8fd995c4-n7v4f"] Jul 23 19:29:19.355206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.355086 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56412035-bff1-456d-95a8-f9e71229f5fe" containerName="manager" Jul 23 19:29:19.355206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.355099 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="56412035-bff1-456d-95a8-f9e71229f5fe" containerName="manager" Jul 23 19:29:19.355206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.355114 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96846a4-0aa9-4136-bdce-72c52e9a0d4c" containerName="authorino" Jul 23 19:29:19.355206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.355119 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96846a4-0aa9-4136-bdce-72c52e9a0d4c" containerName="authorino" Jul 23 19:29:19.355206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.355189 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96846a4-0aa9-4136-bdce-72c52e9a0d4c" containerName="authorino" Jul 23 19:29:19.355206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.355198 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="56412035-bff1-456d-95a8-f9e71229f5fe" containerName="manager" Jul 23 19:29:19.358172 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.358154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.360865 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.360842 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 19:29:19.361619 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.361604 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mfldh\"" Jul 23 19:29:19.361714 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.361659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 19:29:19.369372 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.369344 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-n7v4f"] Jul 23 19:29:19.418145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.418104 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d7c5e85-64ae-4327-9010-97e59be352c3-webhook-cert\") pod \"maas-controller-c8fd995c4-n7v4f\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.418145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.418147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgfpp\" (UniqueName: \"kubernetes.io/projected/2d7c5e85-64ae-4327-9010-97e59be352c3-kube-api-access-zgfpp\") pod \"maas-controller-c8fd995c4-n7v4f\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.519145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.519112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d7c5e85-64ae-4327-9010-97e59be352c3-webhook-cert\") pod \"maas-controller-c8fd995c4-n7v4f\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.519145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.519152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgfpp\" (UniqueName: \"kubernetes.io/projected/2d7c5e85-64ae-4327-9010-97e59be352c3-kube-api-access-zgfpp\") pod \"maas-controller-c8fd995c4-n7v4f\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.521513 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.521490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d7c5e85-64ae-4327-9010-97e59be352c3-webhook-cert\") pod \"maas-controller-c8fd995c4-n7v4f\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.536144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.536116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgfpp\" (UniqueName: \"kubernetes.io/projected/2d7c5e85-64ae-4327-9010-97e59be352c3-kube-api-access-zgfpp\") pod \"maas-controller-c8fd995c4-n7v4f\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.669951 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.669912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:19.811296 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:19.811269 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-n7v4f"] Jul 23 19:29:19.814715 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:29:19.814683 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7c5e85_64ae_4327_9010_97e59be352c3.slice/crio-91a6f7196d589a53ec2ce801f88597d82de97433e3c8c3a0f17745c8712cdced WatchSource:0}: Error finding container 91a6f7196d589a53ec2ce801f88597d82de97433e3c8c3a0f17745c8712cdced: Status 404 returned error can't find the container with id 91a6f7196d589a53ec2ce801f88597d82de97433e3c8c3a0f17745c8712cdced Jul 23 19:29:20.815897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:20.815797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" event={"ID":"2d7c5e85-64ae-4327-9010-97e59be352c3","Type":"ContainerStarted","Data":"270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e"} Jul 23 19:29:20.815897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:20.815851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" event={"ID":"2d7c5e85-64ae-4327-9010-97e59be352c3","Type":"ContainerStarted","Data":"91a6f7196d589a53ec2ce801f88597d82de97433e3c8c3a0f17745c8712cdced"} Jul 23 19:29:20.815897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:20.815882 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:20.833832 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:20.833778 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" podStartSLOduration=1.2117886549999999 podStartE2EDuration="1.833758459s" totalImagesPullingTime="621.969804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:29:20.832323963 +0000 UTC m=+808.404944518" watchObservedRunningTime="2026-07-23 19:29:20.833758459 +0000 UTC m=+808.406379031" Jul 23 19:29:31.822625 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:31.822593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:36.464817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.464775 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-n7v4f"] Jul 23 19:29:36.465234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.464989 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" podUID="2d7c5e85-64ae-4327-9010-97e59be352c3" containerName="manager" containerID="cri-o://270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e" gracePeriod=30 Jul 23 19:29:36.718768 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.718693 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:36.874433 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.874405 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgfpp\" (UniqueName: \"kubernetes.io/projected/2d7c5e85-64ae-4327-9010-97e59be352c3-kube-api-access-zgfpp\") pod \"2d7c5e85-64ae-4327-9010-97e59be352c3\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " Jul 23 19:29:36.874599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.874439 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d7c5e85-64ae-4327-9010-97e59be352c3-webhook-cert\") pod \"2d7c5e85-64ae-4327-9010-97e59be352c3\" (UID: \"2d7c5e85-64ae-4327-9010-97e59be352c3\") " Jul 23 19:29:36.876570 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.876539 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7c5e85-64ae-4327-9010-97e59be352c3-kube-api-access-zgfpp" (OuterVolumeSpecName: "kube-api-access-zgfpp") pod "2d7c5e85-64ae-4327-9010-97e59be352c3" (UID: "2d7c5e85-64ae-4327-9010-97e59be352c3"). InnerVolumeSpecName "kube-api-access-zgfpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:29:36.876570 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.876546 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7c5e85-64ae-4327-9010-97e59be352c3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2d7c5e85-64ae-4327-9010-97e59be352c3" (UID: "2d7c5e85-64ae-4327-9010-97e59be352c3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:29:36.900196 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.900166 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d7c5e85-64ae-4327-9010-97e59be352c3" containerID="270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e" exitCode=0 Jul 23 19:29:36.900355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.900221 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" Jul 23 19:29:36.900355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.900244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" event={"ID":"2d7c5e85-64ae-4327-9010-97e59be352c3","Type":"ContainerDied","Data":"270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e"} Jul 23 19:29:36.900355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.900298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-n7v4f" event={"ID":"2d7c5e85-64ae-4327-9010-97e59be352c3","Type":"ContainerDied","Data":"91a6f7196d589a53ec2ce801f88597d82de97433e3c8c3a0f17745c8712cdced"} Jul 23 19:29:36.900355 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.900319 2574 scope.go:117] "RemoveContainer" containerID="270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e" Jul 23 19:29:36.909919 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.909898 2574 scope.go:117] "RemoveContainer" containerID="270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e" Jul 23 19:29:36.910279 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:29:36.910252 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e\": container with ID starting with 270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e not found: ID does not exist" containerID="270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e" Jul 23 19:29:36.910334 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.910289 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e"} err="failed to get container status \"270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e\": rpc error: code = NotFound desc = could not find container \"270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e\": container with ID starting with 270adfb0686d1a06433d28980b1e9981cf362392f11e97716fc8664b1c96756e not found: ID does not exist" Jul 23 19:29:36.922397 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.922370 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-n7v4f"] Jul 23 19:29:36.926140 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.926116 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-n7v4f"] Jul 23 19:29:36.975726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.975645 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgfpp\" (UniqueName: \"kubernetes.io/projected/2d7c5e85-64ae-4327-9010-97e59be352c3-kube-api-access-zgfpp\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:29:36.975726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:36.975676 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d7c5e85-64ae-4327-9010-97e59be352c3-webhook-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:29:37.718572 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.718539 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8fd995c4-wlzdx"] Jul 23 19:29:37.719044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.718911 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7c5e85-64ae-4327-9010-97e59be352c3" containerName="manager" Jul 23 19:29:37.719044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.718922 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7c5e85-64ae-4327-9010-97e59be352c3" containerName="manager" Jul 23 19:29:37.719044 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.719013 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7c5e85-64ae-4327-9010-97e59be352c3" containerName="manager" Jul 23 19:29:37.723336 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.723316 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:37.725834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.725813 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mfldh\"" Jul 23 19:29:37.726997 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.726984 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 19:29:37.727052 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.727035 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 19:29:37.733860 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.733831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-wlzdx"] Jul 23 19:29:37.883899 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.883857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqxw\" (UniqueName: \"kubernetes.io/projected/7c551bba-efdf-4b2d-a806-f98d7eafa6f9-kube-api-access-ssqxw\") pod \"maas-controller-c8fd995c4-wlzdx\" (UID: \"7c551bba-efdf-4b2d-a806-f98d7eafa6f9\") " pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:37.883899 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.883903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c551bba-efdf-4b2d-a806-f98d7eafa6f9-webhook-cert\") pod \"maas-controller-c8fd995c4-wlzdx\" (UID: \"7c551bba-efdf-4b2d-a806-f98d7eafa6f9\") " pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:37.985413 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.985323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqxw\" (UniqueName: \"kubernetes.io/projected/7c551bba-efdf-4b2d-a806-f98d7eafa6f9-kube-api-access-ssqxw\") pod \"maas-controller-c8fd995c4-wlzdx\" (UID: \"7c551bba-efdf-4b2d-a806-f98d7eafa6f9\") " pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:37.985413 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.985374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c551bba-efdf-4b2d-a806-f98d7eafa6f9-webhook-cert\") pod \"maas-controller-c8fd995c4-wlzdx\" (UID: \"7c551bba-efdf-4b2d-a806-f98d7eafa6f9\") " pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:37.987820 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.987791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c551bba-efdf-4b2d-a806-f98d7eafa6f9-webhook-cert\") pod \"maas-controller-c8fd995c4-wlzdx\" (UID: \"7c551bba-efdf-4b2d-a806-f98d7eafa6f9\") " pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:37.994539 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:37.994509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqxw\" (UniqueName: \"kubernetes.io/projected/7c551bba-efdf-4b2d-a806-f98d7eafa6f9-kube-api-access-ssqxw\") pod \"maas-controller-c8fd995c4-wlzdx\" (UID: \"7c551bba-efdf-4b2d-a806-f98d7eafa6f9\") " pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:38.034526 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.034486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:38.177079 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.177041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8fd995c4-wlzdx"] Jul 23 19:29:38.180394 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:29:38.180361 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c551bba_efdf_4b2d_a806_f98d7eafa6f9.slice/crio-a79e0e56eef1705f2f1ece14b9eb972c1340409cd079d163ab941b1ed8a509c7 WatchSource:0}: Error finding container a79e0e56eef1705f2f1ece14b9eb972c1340409cd079d163ab941b1ed8a509c7: Status 404 returned error can't find the container with id a79e0e56eef1705f2f1ece14b9eb972c1340409cd079d163ab941b1ed8a509c7 Jul 23 19:29:38.847017 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.846985 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7c5e85-64ae-4327-9010-97e59be352c3" path="/var/lib/kubelet/pods/2d7c5e85-64ae-4327-9010-97e59be352c3/volumes" Jul 23 19:29:38.916457 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.916413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" event={"ID":"7c551bba-efdf-4b2d-a806-f98d7eafa6f9","Type":"ContainerStarted","Data":"b25e7776e49ed964330397fc8a1a684e88700f7734e641c4b1c9680eca5a9ef1"} Jul 23 19:29:38.916644 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.916501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" event={"ID":"7c551bba-efdf-4b2d-a806-f98d7eafa6f9","Type":"ContainerStarted","Data":"a79e0e56eef1705f2f1ece14b9eb972c1340409cd079d163ab941b1ed8a509c7"} Jul 23 19:29:38.916644 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.916555 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:29:38.935831 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:38.935778 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" podStartSLOduration=1.404170753 podStartE2EDuration="1.93576045s" totalImagesPullingTime="531.589697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:29:38.934509847 +0000 UTC m=+826.507130387" watchObservedRunningTime="2026-07-23 19:29:38.93576045 +0000 UTC m=+826.508380990" Jul 23 19:29:49.923414 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:29:49.923379 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8fd995c4-wlzdx" Jul 23 19:30:00.142349 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:00.142306 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv"] Jul 23 19:30:00.147806 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:00.147776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" Jul 23 19:30:00.151235 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:00.151207 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwrbz\"" Jul 23 19:30:00.153201 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:00.153166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv"] Jul 23 19:30:00.163164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:00.163095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" Jul 23 19:30:00.565024 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:00.564997 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv"] Jul 23 19:30:00.568339 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:30:00.568292 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93abb7f_11f6_4e4b_9de6_766d5a38ad02.slice/crio-77f475d496230726d9cd29cf196d6dec6794bacd10ddb131cf10ecdf3b894617 WatchSource:0}: Error finding container 77f475d496230726d9cd29cf196d6dec6794bacd10ddb131cf10ecdf3b894617: Status 404 returned error can't find the container with id 77f475d496230726d9cd29cf196d6dec6794bacd10ddb131cf10ecdf3b894617 Jul 23 19:30:01.029790 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:01.029746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" event={"ID":"a93abb7f-11f6-4e4b-9de6-766d5a38ad02","Type":"ContainerStarted","Data":"77f475d496230726d9cd29cf196d6dec6794bacd10ddb131cf10ecdf3b894617"} Jul 23 19:30:05.058835 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:05.058793 2574 generic.go:356] "Generic (PLEG): container finished" podID="a93abb7f-11f6-4e4b-9de6-766d5a38ad02" containerID="8c03f855288eb4a68759a22aabd70ba91cd18e2fa58dab859f14567a314cc4ec" exitCode=0 Jul 23 19:30:05.059207 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:05.058845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" event={"ID":"a93abb7f-11f6-4e4b-9de6-766d5a38ad02","Type":"ContainerDied","Data":"8c03f855288eb4a68759a22aabd70ba91cd18e2fa58dab859f14567a314cc4ec"} Jul 23 19:30:06.207618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:06.207592 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" Jul 23 19:30:07.069951 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:07.069923 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" Jul 23 19:30:07.070117 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:07.069947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747250-h8dxv" event={"ID":"a93abb7f-11f6-4e4b-9de6-766d5a38ad02","Type":"ContainerDied","Data":"77f475d496230726d9cd29cf196d6dec6794bacd10ddb131cf10ecdf3b894617"} Jul 23 19:30:07.070117 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:07.069998 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f475d496230726d9cd29cf196d6dec6794bacd10ddb131cf10ecdf3b894617" Jul 23 19:30:52.826196 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:52.826117 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:30:52.826787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:30:52.826354 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:35:23.994444 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:23.994360 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb"] Jul 23 19:35:23.994921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:23.994801 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93abb7f-11f6-4e4b-9de6-766d5a38ad02" containerName="cleanup" Jul 23 19:35:23.994921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:23.994815 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93abb7f-11f6-4e4b-9de6-766d5a38ad02" containerName="cleanup" Jul 23 19:35:23.994921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:23.994899 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93abb7f-11f6-4e4b-9de6-766d5a38ad02" containerName="cleanup" Jul 23 19:35:23.997920 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:23.997903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.000401 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.000382 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-shfjv\"" Jul 23 19:35:24.010067 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.010044 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb"] Jul 23 19:35:24.101250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.101210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/56a96f9c-a12c-44c4-9395-8215c34a6671-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2sjwb\" (UID: \"56a96f9c-a12c-44c4-9395-8215c34a6671\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.101436 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.101275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwd7\" (UniqueName: \"kubernetes.io/projected/56a96f9c-a12c-44c4-9395-8215c34a6671-kube-api-access-blwd7\") pod \"kuadrant-operator-controller-manager-86878c769f-2sjwb\" (UID: \"56a96f9c-a12c-44c4-9395-8215c34a6671\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.202283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.202226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/56a96f9c-a12c-44c4-9395-8215c34a6671-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2sjwb\" (UID: \"56a96f9c-a12c-44c4-9395-8215c34a6671\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.202480 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.202317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwd7\" (UniqueName: \"kubernetes.io/projected/56a96f9c-a12c-44c4-9395-8215c34a6671-kube-api-access-blwd7\") pod \"kuadrant-operator-controller-manager-86878c769f-2sjwb\" (UID: \"56a96f9c-a12c-44c4-9395-8215c34a6671\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.202670 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.202650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/56a96f9c-a12c-44c4-9395-8215c34a6671-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2sjwb\" (UID: \"56a96f9c-a12c-44c4-9395-8215c34a6671\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.210970 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.210946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwd7\" (UniqueName: \"kubernetes.io/projected/56a96f9c-a12c-44c4-9395-8215c34a6671-kube-api-access-blwd7\") pod \"kuadrant-operator-controller-manager-86878c769f-2sjwb\" (UID: \"56a96f9c-a12c-44c4-9395-8215c34a6671\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.308588 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.308497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.457976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.457936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb"] Jul 23 19:35:24.461388 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:35:24.461358 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a96f9c_a12c_44c4_9395_8215c34a6671.slice/crio-12d23c9212c8542337506edd1a355b8234e722205553d1733f6553b7e1442ab8 WatchSource:0}: Error finding container 12d23c9212c8542337506edd1a355b8234e722205553d1733f6553b7e1442ab8: Status 404 returned error can't find the container with id 12d23c9212c8542337506edd1a355b8234e722205553d1733f6553b7e1442ab8 Jul 23 19:35:24.677663 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.677628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" event={"ID":"56a96f9c-a12c-44c4-9395-8215c34a6671","Type":"ContainerStarted","Data":"2b957ca9d94421f7744077e29bc892ed13eb8c65cb677c7dbc3cb39618d831ce"} Jul 23 19:35:24.677877 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.677681 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" event={"ID":"56a96f9c-a12c-44c4-9395-8215c34a6671","Type":"ContainerStarted","Data":"12d23c9212c8542337506edd1a355b8234e722205553d1733f6553b7e1442ab8"} Jul 23 19:35:24.677930 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.677877 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:24.698107 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:24.698054 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" podStartSLOduration=1.6980348140000001 podStartE2EDuration="1.698034814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:35:24.696867964 +0000 UTC m=+1172.269488504" watchObservedRunningTime="2026-07-23 19:35:24.698034814 +0000 UTC m=+1172.270655354" Jul 23 19:35:35.685453 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:35.685420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sjwb" Jul 23 19:35:52.857348 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:52.857322 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:35:52.859400 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:35:52.859378 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:40:52.886797 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:40:52.886769 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:40:52.890459 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:40:52.890438 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:42:38.921652 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:38.921613 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq"] Jul 23 19:42:38.925180 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:38.925160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:38.927997 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:38.927979 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c543a96f-openshift-default-dockercfg-j5lzn\"" Jul 23 19:42:38.945407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:38.945376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq"] Jul 23 19:42:39.046435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-credential-socket\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqff\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-kube-api-access-7wqff\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046477 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-envoy\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de950b56-e493-46ba-bebe-b6571003e6a5-istio-podinfo\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-istio-token\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-socket\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046618 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de950b56-e493-46ba-bebe-b6571003e6a5-istiod-ca-cert\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-certs\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.046874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.046707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-data\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147394 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-socket\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147394 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de950b56-e493-46ba-bebe-b6571003e6a5-istiod-ca-cert\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-certs\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-data\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-credential-socket\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqff\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-kube-api-access-7wqff\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-envoy\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de950b56-e493-46ba-bebe-b6571003e6a5-istio-podinfo\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.147669 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-istio-token\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.148119 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147909 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-socket\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.148119 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.147999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-certs\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.148232 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.148159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-credential-socket\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.148300 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.148231 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-data\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.148300 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.148280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de950b56-e493-46ba-bebe-b6571003e6a5-istiod-ca-cert\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.149968 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.149940 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-envoy\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.150187 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.150168 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de950b56-e493-46ba-bebe-b6571003e6a5-istio-podinfo\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.155826 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.155794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-istio-token\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.156061 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.156040 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqff\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-kube-api-access-7wqff\") pod \"e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.236176 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.236083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:39.379080 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.379057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq"] Jul 23 19:42:39.381012 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:42:39.380986 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde950b56_e493_46ba_bebe_b6571003e6a5.slice/crio-7ae24b000d996881ab16d6eaad5ed056c0b4c5af4eed96ac117738e64b002f13 WatchSource:0}: Error finding container 7ae24b000d996881ab16d6eaad5ed056c0b4c5af4eed96ac117738e64b002f13: Status 404 returned error can't find the container with id 7ae24b000d996881ab16d6eaad5ed056c0b4c5af4eed96ac117738e64b002f13 Jul 23 19:42:39.383256 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.383221 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:42:39.383361 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.383295 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:42:39.383361 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.383324 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:42:39.987304 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.987268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" event={"ID":"de950b56-e493-46ba-bebe-b6571003e6a5","Type":"ContainerStarted","Data":"f8a5166d3e229563e5b3425347f50ab36ec04f43dfca69530a5d7575d86067e2"} Jul 23 19:42:39.987703 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:39.987354 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" event={"ID":"de950b56-e493-46ba-bebe-b6571003e6a5","Type":"ContainerStarted","Data":"7ae24b000d996881ab16d6eaad5ed056c0b4c5af4eed96ac117738e64b002f13"} Jul 23 19:42:40.010031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:40.009981 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" podStartSLOduration=2.009966151 podStartE2EDuration="2.009966151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:42:40.007898631 +0000 UTC m=+1607.580519175" watchObservedRunningTime="2026-07-23 19:42:40.009966151 +0000 UTC m=+1607.582586690" Jul 23 19:42:40.236808 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:40.236763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:40.242041 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:40.241956 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:40.992054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:40.992016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:40.993007 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:40.992988 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:53.055941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:53.055902 2574 generic.go:356] "Generic (PLEG): container finished" podID="de950b56-e493-46ba-bebe-b6571003e6a5" containerID="f8a5166d3e229563e5b3425347f50ab36ec04f43dfca69530a5d7575d86067e2" exitCode=0 Jul 23 19:42:53.056493 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:53.055966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" event={"ID":"de950b56-e493-46ba-bebe-b6571003e6a5","Type":"ContainerDied","Data":"f8a5166d3e229563e5b3425347f50ab36ec04f43dfca69530a5d7575d86067e2"} Jul 23 19:42:53.056563 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:53.056490 2574 scope.go:117] "RemoveContainer" containerID="f8a5166d3e229563e5b3425347f50ab36ec04f43dfca69530a5d7575d86067e2" Jul 23 19:42:53.057518 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:53.057488 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:42:53.057616 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:53.057559 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:42:53.057616 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:53.057594 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:42:54.064580 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:54.064544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" event={"ID":"de950b56-e493-46ba-bebe-b6571003e6a5","Type":"ContainerStarted","Data":"8032ecc2fb45eb9fb3b048190338013a2563a3192a7fe5d2664a34aa86fbf084"} Jul 23 19:42:54.237320 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:54.237286 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:54.238605 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:54.238570 2574 patch_prober.go:28] interesting pod/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 19:42:54.238848 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:54.238820 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 19:42:55.236919 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:55.236876 2574 patch_prober.go:28] interesting pod/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 19:42:55.237464 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:55.236947 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 19:42:55.992863 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:55.992825 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:56.237300 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:56.237253 2574 patch_prober.go:28] interesting pod/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 19:42:56.237957 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:56.237917 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 19:42:57.236778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:57.236713 2574 patch_prober.go:28] interesting pod/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 23 19:42:57.237109 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:57.237076 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 23 19:42:58.240301 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:58.240273 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:42:58.241424 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:42:58.241401 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:43:22.074165 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:22.074131 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq"] Jul 23 19:43:22.074812 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:22.074399 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" containerID="cri-o://8032ecc2fb45eb9fb3b048190338013a2563a3192a7fe5d2664a34aa86fbf084" gracePeriod=30 Jul 23 19:43:23.069113 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.069080 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr"] Jul 23 19:43:23.072803 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.072784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.075492 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.075470 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e21d0ba1-gw-openshift-default-dockercfg-xbdsl\"" Jul 23 19:43:23.085451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.085419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr"] Jul 23 19:43:23.153890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.153851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-certs\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.153890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.153890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-envoy\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.153922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-credential-socket\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.153963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-podinfo\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.154025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-socket\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.154043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80e5294-cf7a-4400-9636-8ce6ff57e188-istiod-ca-cert\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.154074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zlc\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-kube-api-access-98zlc\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154434 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.154198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-data\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.154434 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.154224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-token\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.254921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.254885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-certs\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.254921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.254926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-envoy\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.254971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-credential-socket\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-podinfo\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-socket\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80e5294-cf7a-4400-9636-8ce6ff57e188-istiod-ca-cert\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255192 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zlc\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-kube-api-access-98zlc\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255463 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-certs\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255562 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-data\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255640 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-socket\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255702 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255652 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-token\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.255941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.255918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-data\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.256043 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.256022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80e5294-cf7a-4400-9636-8ce6ff57e188-istiod-ca-cert\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.256149 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.256128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-credential-socket\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.257871 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.257845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-envoy\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.258202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.258173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-podinfo\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.263563 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.263538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-token\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.263709 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.263686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zlc\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-kube-api-access-98zlc\") pod \"e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.383435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.383348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:23.524773 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.524720 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr"] Jul 23 19:43:23.526651 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:43:23.526621 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80e5294_cf7a_4400_9636_8ce6ff57e188.slice/crio-ee04582719c4ac9dd04eecdaf4252c820a32edb25484efa77eb294321fdd465d WatchSource:0}: Error finding container ee04582719c4ac9dd04eecdaf4252c820a32edb25484efa77eb294321fdd465d: Status 404 returned error can't find the container with id ee04582719c4ac9dd04eecdaf4252c820a32edb25484efa77eb294321fdd465d Jul 23 19:43:23.528651 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.528621 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:43:23.528724 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.528684 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:43:23.528724 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:23.528717 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:43:24.217361 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:24.217326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" event={"ID":"c80e5294-cf7a-4400-9636-8ce6ff57e188","Type":"ContainerStarted","Data":"10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53"} Jul 23 19:43:24.217823 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:24.217381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" event={"ID":"c80e5294-cf7a-4400-9636-8ce6ff57e188","Type":"ContainerStarted","Data":"ee04582719c4ac9dd04eecdaf4252c820a32edb25484efa77eb294321fdd465d"} Jul 23 19:43:24.239123 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:24.239060 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podStartSLOduration=1.239038008 podStartE2EDuration="1.239038008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:43:24.236618952 +0000 UTC m=+1651.809239491" watchObservedRunningTime="2026-07-23 19:43:24.239038008 +0000 UTC m=+1651.811658556" Jul 23 19:43:24.384077 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:24.384045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:24.389248 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:24.389224 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:25.222392 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:25.222359 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:25.223594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:25.223580 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:27.237595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.237565 2574 generic.go:356] "Generic (PLEG): container finished" podID="de950b56-e493-46ba-bebe-b6571003e6a5" containerID="8032ecc2fb45eb9fb3b048190338013a2563a3192a7fe5d2664a34aa86fbf084" exitCode=0 Jul 23 19:43:27.238068 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.237908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" event={"ID":"de950b56-e493-46ba-bebe-b6571003e6a5","Type":"ContainerDied","Data":"8032ecc2fb45eb9fb3b048190338013a2563a3192a7fe5d2664a34aa86fbf084"} Jul 23 19:43:27.238068 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.237982 2574 scope.go:117] "RemoveContainer" containerID="f8a5166d3e229563e5b3425347f50ab36ec04f43dfca69530a5d7575d86067e2" Jul 23 19:43:27.348055 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.347991 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:43:27.496988 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.496958 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-socket\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497014 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-istio-token\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497052 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wqff\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-kube-api-access-7wqff\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497076 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de950b56-e493-46ba-bebe-b6571003e6a5-istio-podinfo\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497109 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-credential-socket\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497397 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-certs\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497397 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497315 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:43:27.497397 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497342 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-envoy\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497397 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497370 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-data\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497414 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:43:27.497594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497472 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de950b56-e493-46ba-bebe-b6571003e6a5-istiod-ca-cert\") pod \"de950b56-e493-46ba-bebe-b6571003e6a5\" (UID: \"de950b56-e493-46ba-bebe-b6571003e6a5\") " Jul 23 19:43:27.497594 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497521 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:43:27.497867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497804 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.497867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497826 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.497867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497841 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.497867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.497848 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de950b56-e493-46ba-bebe-b6571003e6a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:43:27.498405 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.498373 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:43:27.499623 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.499599 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de950b56-e493-46ba-bebe-b6571003e6a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:43:27.499809 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.499792 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:43:27.499939 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.499925 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:43:27.499994 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.499953 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-kube-api-access-7wqff" (OuterVolumeSpecName: "kube-api-access-7wqff") pod "de950b56-e493-46ba-bebe-b6571003e6a5" (UID: "de950b56-e493-46ba-bebe-b6571003e6a5"). InnerVolumeSpecName "kube-api-access-7wqff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:43:27.598767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.598670 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.598767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.598696 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de950b56-e493-46ba-bebe-b6571003e6a5-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.598767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.598705 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de950b56-e493-46ba-bebe-b6571003e6a5-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.598767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.598714 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.598767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.598723 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqff\" (UniqueName: \"kubernetes.io/projected/de950b56-e493-46ba-bebe-b6571003e6a5-kube-api-access-7wqff\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:27.598767 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:27.598744 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de950b56-e493-46ba-bebe-b6571003e6a5-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:43:28.245237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:28.245201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" event={"ID":"de950b56-e493-46ba-bebe-b6571003e6a5","Type":"ContainerDied","Data":"7ae24b000d996881ab16d6eaad5ed056c0b4c5af4eed96ac117738e64b002f13"} Jul 23 19:43:28.245710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:28.245275 2574 scope.go:117] "RemoveContainer" containerID="8032ecc2fb45eb9fb3b048190338013a2563a3192a7fe5d2664a34aa86fbf084" Jul 23 19:43:28.245710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:28.245213 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq" Jul 23 19:43:28.268655 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:28.268628 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq"] Jul 23 19:43:28.272189 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:28.272161 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c543a96f-openshift-default-98bdcd4df-nhxcq"] Jul 23 19:43:28.848020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:28.847986 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" path="/var/lib/kubelet/pods/de950b56-e493-46ba-bebe-b6571003e6a5/volumes" Jul 23 19:43:40.223984 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.223942 2574 patch_prober.go:28] interesting pod/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 19:43:40.224431 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.223998 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 19:43:40.314978 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.314944 2574 generic.go:356] "Generic (PLEG): container finished" podID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerID="10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53" exitCode=0 Jul 23 19:43:40.315159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.315020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" event={"ID":"c80e5294-cf7a-4400-9636-8ce6ff57e188","Type":"ContainerDied","Data":"10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53"} Jul 23 19:43:40.315445 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.315427 2574 scope.go:117] "RemoveContainer" containerID="10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53" Jul 23 19:43:40.316455 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.316422 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:43:40.316552 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.316493 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:43:40.316552 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:40.316527 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:43:41.322591 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:41.322551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" event={"ID":"c80e5294-cf7a-4400-9636-8ce6ff57e188","Type":"ContainerStarted","Data":"d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3"} Jul 23 19:43:41.384255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:41.384212 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:41.384467 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:41.384419 2574 patch_prober.go:28] interesting pod/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 19:43:41.384544 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:41.384494 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 19:43:42.384437 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:42.384393 2574 patch_prober.go:28] interesting pod/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 19:43:42.384871 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:42.384470 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 19:43:43.384317 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:43.384277 2574 patch_prober.go:28] interesting pod/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 19:43:43.384502 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:43.384333 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 19:43:44.384058 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:44.384019 2574 patch_prober.go:28] interesting pod/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 23 19:43:44.384247 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:44.384073 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 23 19:43:45.388335 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:45.388303 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:45.388714 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:45.388602 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:43:45.389452 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:43:45.389432 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:44:05.879348 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:05.879304 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr"] Jul 23 19:44:05.879845 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:05.879595 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" containerID="cri-o://d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3" gracePeriod=30 Jul 23 19:44:06.613316 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.613281 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd"] Jul 23 19:44:06.613778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.613763 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" Jul 23 19:44:06.613838 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.613781 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" Jul 23 19:44:06.613879 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.613848 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" Jul 23 19:44:06.617046 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.617028 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.619542 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.619524 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d3681182-openshift-default-dockercfg-ghxtt\"" Jul 23 19:44:06.628997 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.628961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd"] Jul 23 19:44:06.771762 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.771694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-data\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.771950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.771800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7pg\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-kube-api-access-xv7pg\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.771950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.771842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-envoy\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.771950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.771871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istiod-ca-cert\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.771950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.771930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-socket\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.772173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.772054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-certs\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.772173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.772092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-podinfo\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.772173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.772160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-credential-socket\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.772323 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.772190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-token\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873541 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-socket\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873541 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-certs\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873541 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-podinfo\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-credential-socket\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-token\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-data\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7pg\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-kube-api-access-xv7pg\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-envoy\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.873866 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.873807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istiod-ca-cert\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.874159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.874018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-certs\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.874159 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.874125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-data\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.874263 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.874158 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-credential-socket\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.874409 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.874389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-socket\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.874521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.874506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istiod-ca-cert\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.876092 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.876071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-envoy\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.876092 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.876089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-podinfo\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.883121 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.883094 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-token\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.883405 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.883249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7pg\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-kube-api-access-xv7pg\") pod \"e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:06.927650 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:06.927616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:07.072917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.072888 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd"] Jul 23 19:44:07.075821 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:44:07.075795 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64335dd_1d06_4cb6_9ff4_2a62ee04f0fd.slice/crio-7de01152214d17fc5202f5ebd64421d685cb4d9cd4f9366737e1055bb0420994 WatchSource:0}: Error finding container 7de01152214d17fc5202f5ebd64421d685cb4d9cd4f9366737e1055bb0420994: Status 404 returned error can't find the container with id 7de01152214d17fc5202f5ebd64421d685cb4d9cd4f9366737e1055bb0420994 Jul 23 19:44:07.078016 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.077986 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:07.078096 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.078051 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:07.078096 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.078083 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:07.473494 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.473454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" event={"ID":"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd","Type":"ContainerStarted","Data":"0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098"} Jul 23 19:44:07.473725 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.473520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" event={"ID":"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd","Type":"ContainerStarted","Data":"7de01152214d17fc5202f5ebd64421d685cb4d9cd4f9366737e1055bb0420994"} Jul 23 19:44:07.496833 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.496778 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" podStartSLOduration=1.496761346 podStartE2EDuration="1.496761346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:44:07.494133699 +0000 UTC m=+1695.066754243" watchObservedRunningTime="2026-07-23 19:44:07.496761346 +0000 UTC m=+1695.069381878" Jul 23 19:44:07.927707 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:07.927671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:08.933182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:08.933150 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:09.485135 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:09.485104 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:09.486211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:09.486188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:11.162001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.161971 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:44:11.325253 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325217 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-envoy\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325268 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-certs\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325291 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-socket\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325320 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-podinfo\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325337 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80e5294-cf7a-4400-9636-8ce6ff57e188-istiod-ca-cert\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325385 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-credential-socket\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325447 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98zlc\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-kube-api-access-98zlc\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325465 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-data\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325496 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-token\") pod \"c80e5294-cf7a-4400-9636-8ce6ff57e188\" (UID: \"c80e5294-cf7a-4400-9636-8ce6ff57e188\") " Jul 23 19:44:11.325814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325576 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:11.325814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325658 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:11.325814 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325793 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:11.326129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325953 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.326129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325955 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80e5294-cf7a-4400-9636-8ce6ff57e188-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:44:11.326129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325974 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.326129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.325989 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.326520 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.326492 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-data" (OuterVolumeSpecName: "istio-data") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:11.327626 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.327603 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:11.327760 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.327708 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:44:11.328071 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.328051 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-kube-api-access-98zlc" (OuterVolumeSpecName: "kube-api-access-98zlc") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "kube-api-access-98zlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:44:11.328183 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.328078 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-token" (OuterVolumeSpecName: "istio-token") pod "c80e5294-cf7a-4400-9636-8ce6ff57e188" (UID: "c80e5294-cf7a-4400-9636-8ce6ff57e188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:44:11.427402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.427370 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.427402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.427398 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80e5294-cf7a-4400-9636-8ce6ff57e188-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.427402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.427407 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98zlc\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-kube-api-access-98zlc\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.427648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.427416 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.427648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.427426 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.427648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.427434 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80e5294-cf7a-4400-9636-8ce6ff57e188-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:11.498010 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.497972 2574 generic.go:356] "Generic (PLEG): container finished" podID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerID="d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3" exitCode=0 Jul 23 19:44:11.498182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.498036 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" Jul 23 19:44:11.498182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.498055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" event={"ID":"c80e5294-cf7a-4400-9636-8ce6ff57e188","Type":"ContainerDied","Data":"d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3"} Jul 23 19:44:11.498182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.498115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr" event={"ID":"c80e5294-cf7a-4400-9636-8ce6ff57e188","Type":"ContainerDied","Data":"ee04582719c4ac9dd04eecdaf4252c820a32edb25484efa77eb294321fdd465d"} Jul 23 19:44:11.498182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.498142 2574 scope.go:117] "RemoveContainer" containerID="d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3" Jul 23 19:44:11.507630 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.507608 2574 scope.go:117] "RemoveContainer" containerID="10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53" Jul 23 19:44:11.516056 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.516036 2574 scope.go:117] "RemoveContainer" containerID="d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3" Jul 23 19:44:11.516322 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:44:11.516302 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3\": container with ID starting with d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3 not found: ID does not exist" containerID="d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3" Jul 23 19:44:11.516381 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.516362 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3"} err="failed to get container status \"d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3\": rpc error: code = NotFound desc = could not find container \"d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3\": container with ID starting with d5a1a21571139bddbf4d3037459cc6940106095bf517127ddcdc339624e50fc3 not found: ID does not exist" Jul 23 19:44:11.516434 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.516386 2574 scope.go:117] "RemoveContainer" containerID="10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53" Jul 23 19:44:11.516673 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:44:11.516647 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53\": container with ID starting with 10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53 not found: ID does not exist" containerID="10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53" Jul 23 19:44:11.516812 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.516676 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53"} err="failed to get container status \"10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53\": rpc error: code = NotFound desc = could not find container \"10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53\": container with ID starting with 10b63a63e958eb7c33203f6a2ef6a1c894b20d01972fe95a39c8a4b40c1aae53 not found: ID does not exist" Jul 23 19:44:11.520472 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.520443 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr"] Jul 23 19:44:11.522454 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:11.522434 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e21d0ba1-gw-openshift-default-ccdc46fcf-9rvmr"] Jul 23 19:44:12.849168 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:12.849133 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" path="/var/lib/kubelet/pods/c80e5294-cf7a-4400-9636-8ce6ff57e188/volumes" Jul 23 19:44:23.571695 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:23.571615 2574 generic.go:356] "Generic (PLEG): container finished" podID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerID="0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098" exitCode=0 Jul 23 19:44:23.572108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:23.571683 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" event={"ID":"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd","Type":"ContainerDied","Data":"0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098"} Jul 23 19:44:23.572108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:23.572073 2574 scope.go:117] "RemoveContainer" containerID="0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098" Jul 23 19:44:23.573048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:23.573022 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:23.573125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:23.573084 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:23.573125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:23.573118 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:24.486070 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:24.486034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:24.579021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:24.578987 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" event={"ID":"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd","Type":"ContainerStarted","Data":"2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94"} Jul 23 19:44:24.928647 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:24.928612 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:24.930128 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:24.930091 2574 patch_prober.go:28] interesting pod/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 19:44:24.930285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:24.930160 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 19:44:25.928439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:25.928403 2574 patch_prober.go:28] interesting pod/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 19:44:25.928925 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:25.928456 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 19:44:26.928285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:26.928248 2574 patch_prober.go:28] interesting pod/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 19:44:26.928473 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:26.928311 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 19:44:27.928475 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:27.928439 2574 patch_prober.go:28] interesting pod/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 23 19:44:27.928988 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:27.928493 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 23 19:44:28.932835 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:28.932806 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:28.933230 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:28.933175 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:28.933825 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:28.933807 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:50.819361 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:50.819328 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd"] Jul 23 19:44:50.819832 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:50.819541 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" containerID="cri-o://2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94" gracePeriod=30 Jul 23 19:44:51.569327 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.569293 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp"] Jul 23 19:44:51.569983 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.569953 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" Jul 23 19:44:51.569983 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.569980 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" Jul 23 19:44:51.570261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.569999 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" Jul 23 19:44:51.570261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.570008 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" Jul 23 19:44:51.570261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.570044 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" Jul 23 19:44:51.570261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.570053 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" Jul 23 19:44:51.570261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.570156 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" Jul 23 19:44:51.570261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.570173 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="de950b56-e493-46ba-bebe-b6571003e6a5" containerName="istio-proxy" Jul 23 19:44:51.574199 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.574179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.576933 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.576913 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7db7d650-openshift-default-dockercfg-jm28h\"" Jul 23 19:44:51.586563 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.586541 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp"] Jul 23 19:44:51.624295 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-token\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-credential-socket\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-envoy\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-data\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-certs\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-socket\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-podinfo\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vl5k\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-kube-api-access-4vl5k\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.624585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.624589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istiod-ca-cert\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istiod-ca-cert\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-token\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-credential-socket\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-envoy\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-data\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-certs\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.725857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.725809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-socket\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-podinfo\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-credential-socket\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726158 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-data\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl5k\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-kube-api-access-4vl5k\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726391 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-certs\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726453 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istiod-ca-cert\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.726569 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.726549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-socket\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.728053 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.728034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-envoy\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.728344 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.728327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-podinfo\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.734067 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.734039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-token\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.735023 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.734999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl5k\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-kube-api-access-4vl5k\") pod \"e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:51.885523 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:51.885430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:52.031428 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.031391 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp"] Jul 23 19:44:52.032219 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:44:52.032191 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e77e108_4e6b_4de8_a0f7_f9b69c595aad.slice/crio-83c6afaf2bc4aa82aba999e0ca7a6a020c7f8ce33fc9fdc47e04662e5d79b188 WatchSource:0}: Error finding container 83c6afaf2bc4aa82aba999e0ca7a6a020c7f8ce33fc9fdc47e04662e5d79b188: Status 404 returned error can't find the container with id 83c6afaf2bc4aa82aba999e0ca7a6a020c7f8ce33fc9fdc47e04662e5d79b188 Jul 23 19:44:52.034469 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.034432 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:52.034535 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.034513 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:52.034583 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.034561 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:44:52.726629 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.726596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" event={"ID":"2e77e108-4e6b-4de8-a0f7-f9b69c595aad","Type":"ContainerStarted","Data":"8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758"} Jul 23 19:44:52.726822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.726643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" event={"ID":"2e77e108-4e6b-4de8-a0f7-f9b69c595aad","Type":"ContainerStarted","Data":"83c6afaf2bc4aa82aba999e0ca7a6a020c7f8ce33fc9fdc47e04662e5d79b188"} Jul 23 19:44:52.748512 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.748461 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" podStartSLOduration=1.748445536 podStartE2EDuration="1.748445536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:44:52.74579035 +0000 UTC m=+1740.318410890" watchObservedRunningTime="2026-07-23 19:44:52.748445536 +0000 UTC m=+1740.321066075" Jul 23 19:44:52.885629 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:52.885584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:53.891485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:53.891458 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:54.737558 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:54.737524 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:54.738586 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:54.738568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:44:56.081864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.081797 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:56.167353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167315 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv7pg\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-kube-api-access-xv7pg\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167353 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167360 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-certs\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-data\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167412 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-podinfo\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167429 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-envoy\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167444 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-socket\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-token\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167573 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istiod-ca-cert\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167909 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167599 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-credential-socket\") pod \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\" (UID: \"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd\") " Jul 23 19:44:56.167909 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.167861 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:56.168099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.168074 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:56.168229 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.168110 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:56.168229 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.168169 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:44:56.168628 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.168595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:56.169723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.169694 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-kube-api-access-xv7pg" (OuterVolumeSpecName: "kube-api-access-xv7pg") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "kube-api-access-xv7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:44:56.169858 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.169842 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:44:56.169966 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.169945 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:44:56.169966 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.169958 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" (UID: "e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:44:56.268659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268627 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268655 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268665 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268928 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268674 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv7pg\" (UniqueName: \"kubernetes.io/projected/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-kube-api-access-xv7pg\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268928 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268683 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268928 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268691 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268928 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268698 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268928 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268705 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.268928 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.268713 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:44:56.749722 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.749684 2574 generic.go:356] "Generic (PLEG): container finished" podID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerID="2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94" exitCode=0 Jul 23 19:44:56.749940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.749758 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" Jul 23 19:44:56.749940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.749773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" event={"ID":"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd","Type":"ContainerDied","Data":"2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94"} Jul 23 19:44:56.749940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.749832 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd" event={"ID":"e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd","Type":"ContainerDied","Data":"7de01152214d17fc5202f5ebd64421d685cb4d9cd4f9366737e1055bb0420994"} Jul 23 19:44:56.749940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.749855 2574 scope.go:117] "RemoveContainer" containerID="2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94" Jul 23 19:44:56.759451 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.759435 2574 scope.go:117] "RemoveContainer" containerID="0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098" Jul 23 19:44:56.770894 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.770874 2574 scope.go:117] "RemoveContainer" containerID="2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94" Jul 23 19:44:56.771498 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:44:56.771475 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94\": container with ID starting with 2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94 not found: ID does not exist" containerID="2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94" Jul 23 19:44:56.771599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.771505 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94"} err="failed to get container status \"2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94\": rpc error: code = NotFound desc = could not find container \"2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94\": container with ID starting with 2a518d3c6abf6691d12b8703929cb2a16b3990be8daa380e11908eeb6fd74d94 not found: ID does not exist" Jul 23 19:44:56.771599 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.771523 2574 scope.go:117] "RemoveContainer" containerID="0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098" Jul 23 19:44:56.771829 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:44:56.771805 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098\": container with ID starting with 0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098 not found: ID does not exist" containerID="0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098" Jul 23 19:44:56.771893 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.771838 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098"} err="failed to get container status \"0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098\": rpc error: code = NotFound desc = could not find container \"0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098\": container with ID starting with 0797250fe9b4a69cbf084c3b6a47087b6b25363a7ecac1591ad5e07987dbb098 not found: ID does not exist" Jul 23 19:44:56.773558 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.773537 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd"] Jul 23 19:44:56.778917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.778893 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d3681182-openshift-default-7d949d4b45-t7fxd"] Jul 23 19:44:56.847251 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:44:56.847221 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" path="/var/lib/kubelet/pods/e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd/volumes" Jul 23 19:45:07.811975 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:07.811934 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerID="8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758" exitCode=0 Jul 23 19:45:07.812476 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:07.812012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" event={"ID":"2e77e108-4e6b-4de8-a0f7-f9b69c595aad","Type":"ContainerDied","Data":"8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758"} Jul 23 19:45:07.812476 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:07.812410 2574 scope.go:117] "RemoveContainer" containerID="8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758" Jul 23 19:45:07.813364 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:07.813337 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:45:07.813415 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:07.813402 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:45:07.813447 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:07.813434 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:45:08.818880 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:08.818844 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" event={"ID":"2e77e108-4e6b-4de8-a0f7-f9b69c595aad","Type":"ContainerStarted","Data":"669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a"} Jul 23 19:45:08.885688 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:08.885642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:45:08.886897 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:08.886870 2574 patch_prober.go:28] interesting pod/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 23 19:45:08.887027 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:08.886927 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 23 19:45:09.738081 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:09.738040 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:45:09.886079 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:09.886037 2574 patch_prober.go:28] interesting pod/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 23 19:45:09.886460 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:09.886108 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 23 19:45:10.885885 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:10.885847 2574 patch_prober.go:28] interesting pod/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 23 19:45:10.886089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:10.885906 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 23 19:45:11.886127 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:11.886092 2574 patch_prober.go:28] interesting pod/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 23 19:45:11.886520 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:11.886150 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 23 19:45:12.890280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:12.890248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:45:12.891389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:12.891367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:45:34.157879 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:34.157841 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp"] Jul 23 19:45:34.158428 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:34.158098 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" containerID="cri-o://669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a" gracePeriod=30 Jul 23 19:45:39.427385 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.427290 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:45:39.465807 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.465776 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-token\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.465838 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vl5k\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-kube-api-access-4vl5k\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.465917 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-data\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.465945 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istiod-ca-cert\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.465984 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-envoy\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466008 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-socket\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466032 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-credential-socket\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466370 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466083 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-podinfo\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466370 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466107 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-certs\") pod \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\" (UID: \"2e77e108-4e6b-4de8-a0f7-f9b69c595aad\") " Jul 23 19:45:39.466867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466569 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:45:39.466867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466652 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:45:39.466867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466820 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:45:39.467091 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466866 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:45:39.467091 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.466958 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:45:39.468582 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.468554 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:45:39.468707 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.468637 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-kube-api-access-4vl5k" (OuterVolumeSpecName: "kube-api-access-4vl5k") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "kube-api-access-4vl5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:45:39.468972 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.468941 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:45:39.469087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.469074 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e77e108-4e6b-4de8-a0f7-f9b69c595aad" (UID: "2e77e108-4e6b-4de8-a0f7-f9b69c595aad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:45:39.567950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567918 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.567950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567946 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.567950 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567955 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.568252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567963 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.568252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567973 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.568252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567982 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.568252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567990 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.568252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.567998 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.568252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.568006 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vl5k\" (UniqueName: \"kubernetes.io/projected/2e77e108-4e6b-4de8-a0f7-f9b69c595aad-kube-api-access-4vl5k\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:45:39.984796 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.984763 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerID="669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a" exitCode=0 Jul 23 19:45:39.984976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.984854 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" Jul 23 19:45:39.984976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.984895 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" event={"ID":"2e77e108-4e6b-4de8-a0f7-f9b69c595aad","Type":"ContainerDied","Data":"669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a"} Jul 23 19:45:39.984976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.984947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp" event={"ID":"2e77e108-4e6b-4de8-a0f7-f9b69c595aad","Type":"ContainerDied","Data":"83c6afaf2bc4aa82aba999e0ca7a6a020c7f8ce33fc9fdc47e04662e5d79b188"} Jul 23 19:45:39.984976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.984968 2574 scope.go:117] "RemoveContainer" containerID="669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a" Jul 23 19:45:39.994493 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:39.994476 2574 scope.go:117] "RemoveContainer" containerID="8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758" Jul 23 19:45:40.002611 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.002594 2574 scope.go:117] "RemoveContainer" containerID="669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a" Jul 23 19:45:40.002936 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:45:40.002918 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a\": container with ID starting with 669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a not found: ID does not exist" containerID="669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a" Jul 23 19:45:40.002990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.002945 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a"} err="failed to get container status \"669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a\": rpc error: code = NotFound desc = could not find container \"669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a\": container with ID starting with 669afd0b1458b525db3afb49b654103d84362328bcd7ba96b0f24fb60a32774a not found: ID does not exist" Jul 23 19:45:40.002990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.002962 2574 scope.go:117] "RemoveContainer" containerID="8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758" Jul 23 19:45:40.003189 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:45:40.003169 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758\": container with ID starting with 8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758 not found: ID does not exist" containerID="8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758" Jul 23 19:45:40.003234 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.003196 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758"} err="failed to get container status \"8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758\": rpc error: code = NotFound desc = could not find container \"8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758\": container with ID starting with 8d61a32cb3f7c46ed928d8157825cf80b8171053f78d9e219980619db40cd758 not found: ID does not exist" Jul 23 19:45:40.006917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.006897 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp"] Jul 23 19:45:40.012419 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.012400 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7db7d650-openshift-default-cf67f5447-k9cwp"] Jul 23 19:45:40.850755 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:40.850703 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" path="/var/lib/kubelet/pods/2e77e108-4e6b-4de8-a0f7-f9b69c595aad/volumes" Jul 23 19:45:52.917401 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:52.917373 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:45:52.925447 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:45:52.925423 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:47:10.299812 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.299779 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8"] Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300165 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300176 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300184 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300189 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300200 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300205 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300219 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300225 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300289 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300297 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e77e108-4e6b-4de8-a0f7-f9b69c595aad" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300307 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80e5294-cf7a-4400-9636-8ce6ff57e188" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300313 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" Jul 23 19:47:10.300358 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.300319 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64335dd-1d06-4cb6-9ff4-2a62ee04f0fd" containerName="istio-proxy" Jul 23 19:47:10.303502 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.303483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.306182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.306158 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3c342296-openshift-default-dockercfg-lcx4z\"" Jul 23 19:47:10.326248 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.326205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8"] Jul 23 19:47:10.407450 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-credential-socket\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407450 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-envoy\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407718 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-data\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407718 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-podinfo\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407718 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksc2d\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-kube-api-access-ksc2d\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407718 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-socket\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407718 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-token\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407926 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istiod-ca-cert\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.407926 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.407832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-certs\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.508982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-certs\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-credential-socket\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-envoy\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-data\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-podinfo\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksc2d\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-kube-api-access-ksc2d\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-socket\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509553 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-token\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509553 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istiod-ca-cert\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509553 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-certs\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509716 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-credential-socket\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509716 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509596 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-data\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.509857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.509809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-socket\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.510266 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.510244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istiod-ca-cert\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.511567 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.511536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-envoy\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.511955 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.511932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-podinfo\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.522166 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.522051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-token\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.522312 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.522272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksc2d\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-kube-api-access-ksc2d\") pod \"e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.613716 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.613615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:10.771484 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.771457 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8"] Jul 23 19:47:10.773306 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:47:10.773276 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9512fa_e4ea_4cb0_b5f6_18fd9a1307b1.slice/crio-4458dc2f5c0c64bcf400d5d50a3f61637e03beeda9dab49b88c7fc7912b7c0f5 WatchSource:0}: Error finding container 4458dc2f5c0c64bcf400d5d50a3f61637e03beeda9dab49b88c7fc7912b7c0f5: Status 404 returned error can't find the container with id 4458dc2f5c0c64bcf400d5d50a3f61637e03beeda9dab49b88c7fc7912b7c0f5 Jul 23 19:47:10.775956 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.775924 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:47:10.776027 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.775994 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:47:10.776027 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:10.776023 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:47:11.480527 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:11.480484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" event={"ID":"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1","Type":"ContainerStarted","Data":"1d3fc2ea4f1888a133e5a83830636aee8a2f61c6875f2f3610c012735fdef457"} Jul 23 19:47:11.480956 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:11.480546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" event={"ID":"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1","Type":"ContainerStarted","Data":"4458dc2f5c0c64bcf400d5d50a3f61637e03beeda9dab49b88c7fc7912b7c0f5"} Jul 23 19:47:11.501075 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:11.501027 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podStartSLOduration=1.5010114030000001 podStartE2EDuration="1.501011403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:47:11.500003679 +0000 UTC m=+1879.072624219" watchObservedRunningTime="2026-07-23 19:47:11.501011403 +0000 UTC m=+1879.073631943" Jul 23 19:47:11.614721 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:11.614682 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:11.620792 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:11.620765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:12.487148 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:12.487114 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:12.488261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:12.488237 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:28.076595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.075514 2574 patch_prober.go:28] interesting pod/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:38418->10.132.0.57:15021: read: connection reset by peer" start-of-body= Jul 23 19:47:28.076595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.075602 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:38418->10.132.0.57:15021: read: connection reset by peer" Jul 23 19:47:28.590485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.590453 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerID="1d3fc2ea4f1888a133e5a83830636aee8a2f61c6875f2f3610c012735fdef457" exitCode=0 Jul 23 19:47:28.590662 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.590503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" event={"ID":"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1","Type":"ContainerDied","Data":"1d3fc2ea4f1888a133e5a83830636aee8a2f61c6875f2f3610c012735fdef457"} Jul 23 19:47:28.590918 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.590904 2574 scope.go:117] "RemoveContainer" containerID="1d3fc2ea4f1888a133e5a83830636aee8a2f61c6875f2f3610c012735fdef457" Jul 23 19:47:28.591963 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.591931 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:47:28.592082 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.592035 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:47:28.592147 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:28.592084 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:47:29.598396 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:29.598353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" event={"ID":"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1","Type":"ContainerStarted","Data":"20086a75ff6038d8d3455165658e1b643d8386ed1cca107742784915d061d5fd"} Jul 23 19:47:29.613975 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:29.613942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:29.614207 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:29.614188 2574 patch_prober.go:28] interesting pod/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 19:47:29.614285 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:29.614242 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 19:47:30.614179 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:30.614141 2574 patch_prober.go:28] interesting pod/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 19:47:30.614579 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:30.614207 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 19:47:31.614914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:31.614881 2574 patch_prober.go:28] interesting pod/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 19:47:31.615311 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:31.614936 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 19:47:32.614603 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:32.614570 2574 patch_prober.go:28] interesting pod/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 23 19:47:32.614828 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:32.614619 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 23 19:47:33.617760 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:33.617713 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:33.618164 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:33.618079 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:33.618906 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:33.618885 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:46.536173 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:46.536136 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8"] Jul 23 19:47:46.536656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:46.536402 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" containerID="cri-o://20086a75ff6038d8d3455165658e1b643d8386ed1cca107742784915d061d5fd" gracePeriod=30 Jul 23 19:47:48.621095 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:48.621055 2574 patch_prober.go:28] interesting pod/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 19:47:48.621504 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:48.621116 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 19:47:51.714013 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.713982 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerID="20086a75ff6038d8d3455165658e1b643d8386ed1cca107742784915d061d5fd" exitCode=0 Jul 23 19:47:51.714485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.714058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" event={"ID":"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1","Type":"ContainerDied","Data":"20086a75ff6038d8d3455165658e1b643d8386ed1cca107742784915d061d5fd"} Jul 23 19:47:51.714485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.714139 2574 scope.go:117] "RemoveContainer" containerID="1d3fc2ea4f1888a133e5a83830636aee8a2f61c6875f2f3610c012735fdef457" Jul 23 19:47:51.800444 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.800380 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:51.899387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899341 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-podinfo\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899387 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899384 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istiod-ca-cert\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899413 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-data\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899460 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-credential-socket\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899500 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-socket\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899515 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-token\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899549 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-envoy\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899567 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-certs\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.899668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksc2d\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-kube-api-access-ksc2d\") pod \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\" (UID: \"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1\") " Jul 23 19:47:51.900089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899827 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:47:51.900089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:47:51.900089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899975 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:51.900089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.899996 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:51.900089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.900036 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:47:51.900389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.900277 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:47:51.900726 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.900702 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:47:51.901922 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.901893 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:47:51.902040 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.901955 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:47:51.902218 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.902196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:47:51.902297 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:51.902281 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-kube-api-access-ksc2d" (OuterVolumeSpecName: "kube-api-access-ksc2d") pod "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" (UID: "8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1"). InnerVolumeSpecName "kube-api-access-ksc2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:47:52.001145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001091 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.001145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001134 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.001145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001147 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.001145 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001157 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.001431 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001168 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.001431 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001178 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.001431 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.001190 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksc2d\" (UniqueName: \"kubernetes.io/projected/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1-kube-api-access-ksc2d\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:47:52.721886 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.721847 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" Jul 23 19:47:52.721886 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.721882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8" event={"ID":"8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1","Type":"ContainerDied","Data":"4458dc2f5c0c64bcf400d5d50a3f61637e03beeda9dab49b88c7fc7912b7c0f5"} Jul 23 19:47:52.722458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.721941 2574 scope.go:117] "RemoveContainer" containerID="20086a75ff6038d8d3455165658e1b643d8386ed1cca107742784915d061d5fd" Jul 23 19:47:52.744910 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.744878 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8"] Jul 23 19:47:52.749365 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.749339 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3c342296-openshift-default-8c99b8f47-fprd8"] Jul 23 19:47:52.848842 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:47:52.848807 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" path="/var/lib/kubelet/pods/8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1/volumes" Jul 23 19:48:05.728512 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.728479 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4"] Jul 23 19:48:05.729028 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.729004 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" Jul 23 19:48:05.729028 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.729029 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" Jul 23 19:48:05.729188 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.729139 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" Jul 23 19:48:05.729188 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.729148 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" Jul 23 19:48:05.732682 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.732661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.735238 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.735219 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f275417d-openshift-default-dockercfg-smwtg\"" Jul 23 19:48:05.745314 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.745289 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4"] Jul 23 19:48:05.827683 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.827647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-credential-socket\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.827683 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.827683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-certs\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.827954 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.827706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-data\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.827954 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.827867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-podinfo\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.827954 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.827910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-token\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.828096 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.827969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a20541-9cc9-4799-8208-4bad7dc04880-istiod-ca-cert\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.828154 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.828087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-socket\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.828211 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.828162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-envoy\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.828269 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.828218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qrg\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-kube-api-access-w5qrg\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929542 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a20541-9cc9-4799-8208-4bad7dc04880-istiod-ca-cert\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-socket\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-envoy\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qrg\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-kube-api-access-w5qrg\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-credential-socket\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-certs\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.929787 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-data\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-podinfo\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-token\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.929988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-socket\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930271 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.930228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-certs\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930334 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.930269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-data\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930413 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.930389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a20541-9cc9-4799-8208-4bad7dc04880-istiod-ca-cert\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.930456 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.930415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-credential-socket\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.932225 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.932202 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-envoy\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.932452 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.932432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-podinfo\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.937981 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.937951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qrg\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-kube-api-access-w5qrg\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:05.938108 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:05.937965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-token\") pod \"e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:06.043557 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.043471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:06.190797 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.190767 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4"] Jul 23 19:48:06.192003 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:48:06.191978 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a20541_9cc9_4799_8208_4bad7dc04880.slice/crio-864b1aa7e536df2cbbd7104b069df2963e301f8791d75ea9b27da98e063d4543 WatchSource:0}: Error finding container 864b1aa7e536df2cbbd7104b069df2963e301f8791d75ea9b27da98e063d4543: Status 404 returned error can't find the container with id 864b1aa7e536df2cbbd7104b069df2963e301f8791d75ea9b27da98e063d4543 Jul 23 19:48:06.194485 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.194445 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:48:06.194564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.194543 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:48:06.194621 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.194604 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:48:06.799143 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.799098 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" event={"ID":"c2a20541-9cc9-4799-8208-4bad7dc04880","Type":"ContainerStarted","Data":"2b0a742ec77aedb7ff4dd1ff38e265dffe6918372e4267c384ef0527bd229e04"} Jul 23 19:48:06.799538 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.799202 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" event={"ID":"c2a20541-9cc9-4799-8208-4bad7dc04880","Type":"ContainerStarted","Data":"864b1aa7e536df2cbbd7104b069df2963e301f8791d75ea9b27da98e063d4543"} Jul 23 19:48:06.819822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:06.819767 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" podStartSLOduration=1.819720186 podStartE2EDuration="1.819720186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:48:06.818097008 +0000 UTC m=+1934.390717548" watchObservedRunningTime="2026-07-23 19:48:06.819720186 +0000 UTC m=+1934.392340797" Jul 23 19:48:07.044089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:07.044046 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:07.049458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:07.049382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:07.804881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:07.804849 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:07.805914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:07.805900 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:19.272648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.272613 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n"] Jul 23 19:48:19.273158 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.273087 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" Jul 23 19:48:19.273158 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.273102 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9512fa-e4ea-4cb0-b5f6-18fd9a1307b1" containerName="istio-proxy" Jul 23 19:48:19.278695 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.278671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.281624 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.281591 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f275417d\"/\"default-dockercfg-kzwxh\"" Jul 23 19:48:19.281788 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.281595 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f275417d\"/\"kube-root-ca.crt\"" Jul 23 19:48:19.283233 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.283211 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f275417d\"/\"e2e-lifecycle-model-f275417d-kserve-self-signed-certs\"" Jul 23 19:48:19.283536 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.283350 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f275417d\"/\"openshift-service-ca.crt\"" Jul 23 19:48:19.286664 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.286610 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n"] Jul 23 19:48:19.362595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.362553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-home\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.362595 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.362595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-dshm\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.362881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.362634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-model-cache\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.362881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.362704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndvl\" (UniqueName: \"kubernetes.io/projected/47ce3d38-5567-4db1-be1d-a9f751df83c5-kube-api-access-sndvl\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.362881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.362764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ce3d38-5567-4db1-be1d-a9f751df83c5-tls-certs\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.362881 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.362846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-tmp-dir\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.463822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.463784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-model-cache\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464008 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.463844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndvl\" (UniqueName: \"kubernetes.io/projected/47ce3d38-5567-4db1-be1d-a9f751df83c5-kube-api-access-sndvl\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464008 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.463879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ce3d38-5567-4db1-be1d-a9f751df83c5-tls-certs\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464008 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.463917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-tmp-dir\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464008 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.463987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-home\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464008 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.464007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-dshm\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464317 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.464224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-model-cache\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464317 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.464292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-tmp-dir\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.464562 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.464535 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-home\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.466282 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.466253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-dshm\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.466640 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.466615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ce3d38-5567-4db1-be1d-a9f751df83c5-tls-certs\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.471521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.471495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndvl\" (UniqueName: \"kubernetes.io/projected/47ce3d38-5567-4db1-be1d-a9f751df83c5-kube-api-access-sndvl\") pod \"e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.594332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.594240 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:19.741000 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.740972 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n"] Jul 23 19:48:19.743285 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:48:19.743254 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ce3d38_5567_4db1_be1d_a9f751df83c5.slice/crio-13bff9e3111f263824df909f5ca52283e4f81c76ccfb3244baa3249ab294412e WatchSource:0}: Error finding container 13bff9e3111f263824df909f5ca52283e4f81c76ccfb3244baa3249ab294412e: Status 404 returned error can't find the container with id 13bff9e3111f263824df909f5ca52283e4f81c76ccfb3244baa3249ab294412e Jul 23 19:48:19.745316 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.745296 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:48:19.877649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:19.877562 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" event={"ID":"47ce3d38-5567-4db1-be1d-a9f751df83c5","Type":"ContainerStarted","Data":"13bff9e3111f263824df909f5ca52283e4f81c76ccfb3244baa3249ab294412e"} Jul 23 19:48:21.891280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.891251 2574 generic.go:356] "Generic (PLEG): container finished" podID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerID="2b0a742ec77aedb7ff4dd1ff38e265dffe6918372e4267c384ef0527bd229e04" exitCode=0 Jul 23 19:48:21.891758 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.891336 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" event={"ID":"c2a20541-9cc9-4799-8208-4bad7dc04880","Type":"ContainerDied","Data":"2b0a742ec77aedb7ff4dd1ff38e265dffe6918372e4267c384ef0527bd229e04"} Jul 23 19:48:21.891869 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.891854 2574 scope.go:117] "RemoveContainer" containerID="2b0a742ec77aedb7ff4dd1ff38e265dffe6918372e4267c384ef0527bd229e04" Jul 23 19:48:21.892829 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.892799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" event={"ID":"47ce3d38-5567-4db1-be1d-a9f751df83c5","Type":"ContainerStarted","Data":"5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3"} Jul 23 19:48:21.892932 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.892866 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:48:21.892983 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.892933 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:48:21.892983 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.892965 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:48:21.929090 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:21.929039 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" podStartSLOduration=1.723890907 podStartE2EDuration="2.929021597s" totalImagesPullingTime="1.20513069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:48:21.927753553 +0000 UTC m=+1949.500374091" watchObservedRunningTime="2026-07-23 19:48:21.929021597 +0000 UTC m=+1949.501642137" Jul 23 19:48:22.804941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:22.804905 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:22.900977 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:22.900933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" event={"ID":"c2a20541-9cc9-4799-8208-4bad7dc04880","Type":"ContainerStarted","Data":"c6d41a6345f9f45e696fbd8e28bdc6847bbb6e06b4a8bb8666cebc1f01f4497b"} Jul 23 19:48:23.044917 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:23.044870 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:23.046269 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:23.046230 2574 patch_prober.go:28] interesting pod/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 23 19:48:23.046429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:23.046315 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 23 19:48:24.044422 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:24.044373 2574 patch_prober.go:28] interesting pod/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 23 19:48:24.044959 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:24.044447 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 23 19:48:25.045155 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:25.044655 2574 patch_prober.go:28] interesting pod/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 23 19:48:25.045155 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:25.044722 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 23 19:48:26.044870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:26.044830 2574 patch_prober.go:28] interesting pod/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 23 19:48:26.045179 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:26.045151 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 23 19:48:27.048102 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:27.048071 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:27.048581 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:27.048383 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:27.049058 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:27.049039 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:48:29.595283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:29.595240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:29.595832 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:29.595307 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:29.608426 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:29.608396 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:48:29.949821 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:48:29.949793 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:49:18.854501 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:18.854466 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n"] Jul 23 19:49:33.875362 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:33.875309 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" podUID="47ce3d38-5567-4db1-be1d-a9f751df83c5" containerName="main" containerID="cri-o://5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3" gracePeriod=45 Jul 23 19:49:34.137675 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.137610 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:49:34.167649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167619 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-dshm\") pod \"47ce3d38-5567-4db1-be1d-a9f751df83c5\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " Jul 23 19:49:34.167857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167683 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sndvl\" (UniqueName: \"kubernetes.io/projected/47ce3d38-5567-4db1-be1d-a9f751df83c5-kube-api-access-sndvl\") pod \"47ce3d38-5567-4db1-be1d-a9f751df83c5\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " Jul 23 19:49:34.167857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167722 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-home\") pod \"47ce3d38-5567-4db1-be1d-a9f751df83c5\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " Jul 23 19:49:34.167857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167788 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-tmp-dir\") pod \"47ce3d38-5567-4db1-be1d-a9f751df83c5\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " Jul 23 19:49:34.167857 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167838 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ce3d38-5567-4db1-be1d-a9f751df83c5-tls-certs\") pod \"47ce3d38-5567-4db1-be1d-a9f751df83c5\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " Jul 23 19:49:34.168144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-model-cache\") pod \"47ce3d38-5567-4db1-be1d-a9f751df83c5\" (UID: \"47ce3d38-5567-4db1-be1d-a9f751df83c5\") " Jul 23 19:49:34.168144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.167994 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-home" (OuterVolumeSpecName: "home") pod "47ce3d38-5567-4db1-be1d-a9f751df83c5" (UID: "47ce3d38-5567-4db1-be1d-a9f751df83c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:49:34.168266 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.168200 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "47ce3d38-5567-4db1-be1d-a9f751df83c5" (UID: "47ce3d38-5567-4db1-be1d-a9f751df83c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:49:34.168266 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.168237 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-home\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:49:34.168350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.168312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47ce3d38-5567-4db1-be1d-a9f751df83c5" (UID: "47ce3d38-5567-4db1-be1d-a9f751df83c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:49:34.170082 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.170057 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-dshm" (OuterVolumeSpecName: "dshm") pod "47ce3d38-5567-4db1-be1d-a9f751df83c5" (UID: "47ce3d38-5567-4db1-be1d-a9f751df83c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:49:34.170242 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.170222 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ce3d38-5567-4db1-be1d-a9f751df83c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47ce3d38-5567-4db1-be1d-a9f751df83c5" (UID: "47ce3d38-5567-4db1-be1d-a9f751df83c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:49:34.170302 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.170284 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ce3d38-5567-4db1-be1d-a9f751df83c5-kube-api-access-sndvl" (OuterVolumeSpecName: "kube-api-access-sndvl") pod "47ce3d38-5567-4db1-be1d-a9f751df83c5" (UID: "47ce3d38-5567-4db1-be1d-a9f751df83c5"). InnerVolumeSpecName "kube-api-access-sndvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:49:34.268684 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.268648 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-dshm\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:49:34.268684 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.268677 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sndvl\" (UniqueName: \"kubernetes.io/projected/47ce3d38-5567-4db1-be1d-a9f751df83c5-kube-api-access-sndvl\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:49:34.268684 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.268689 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-tmp-dir\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:49:34.269025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.268698 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ce3d38-5567-4db1-be1d-a9f751df83c5-tls-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:49:34.269025 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.268707 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ce3d38-5567-4db1-be1d-a9f751df83c5-model-cache\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:49:34.282569 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.282540 2574 generic.go:356] "Generic (PLEG): container finished" podID="47ce3d38-5567-4db1-be1d-a9f751df83c5" containerID="5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3" exitCode=0 Jul 23 19:49:34.282764 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.282587 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" Jul 23 19:49:34.282764 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.282623 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" event={"ID":"47ce3d38-5567-4db1-be1d-a9f751df83c5","Type":"ContainerDied","Data":"5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3"} Jul 23 19:49:34.282764 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.282680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n" event={"ID":"47ce3d38-5567-4db1-be1d-a9f751df83c5","Type":"ContainerDied","Data":"13bff9e3111f263824df909f5ca52283e4f81c76ccfb3244baa3249ab294412e"} Jul 23 19:49:34.282764 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.282703 2574 scope.go:117] "RemoveContainer" containerID="5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3" Jul 23 19:49:34.292721 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.292696 2574 scope.go:117] "RemoveContainer" containerID="5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3" Jul 23 19:49:34.293056 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:49:34.293039 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3\": container with ID starting with 5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3 not found: ID does not exist" containerID="5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3" Jul 23 19:49:34.293117 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.293068 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3"} err="failed to get container status \"5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3\": rpc error: code = NotFound desc = could not find container \"5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3\": container with ID starting with 5feb0b4d8401df78655b6d33a4ed21003f7e36c67237483b632000eb993c57f3 not found: ID does not exist" Jul 23 19:49:34.304699 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.304663 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n"] Jul 23 19:49:34.307780 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.307725 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f275417d/e2e-lifecycle-model-f275417d-kserve-7cdc8b6c6c-4bh6n"] Jul 23 19:49:34.848375 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:49:34.848339 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ce3d38-5567-4db1-be1d-a9f751df83c5" path="/var/lib/kubelet/pods/47ce3d38-5567-4db1-be1d-a9f751df83c5/volumes" Jul 23 19:50:02.219391 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:02.219356 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4"] Jul 23 19:50:02.219864 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:02.219610 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" containerID="cri-o://c6d41a6345f9f45e696fbd8e28bdc6847bbb6e06b4a8bb8666cebc1f01f4497b" gracePeriod=30 Jul 23 19:50:07.461569 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.461514 2574 generic.go:356] "Generic (PLEG): container finished" podID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerID="c6d41a6345f9f45e696fbd8e28bdc6847bbb6e06b4a8bb8666cebc1f01f4497b" exitCode=0 Jul 23 19:50:07.461979 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.461633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" event={"ID":"c2a20541-9cc9-4799-8208-4bad7dc04880","Type":"ContainerDied","Data":"c6d41a6345f9f45e696fbd8e28bdc6847bbb6e06b4a8bb8666cebc1f01f4497b"} Jul 23 19:50:07.461979 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.461697 2574 scope.go:117] "RemoveContainer" containerID="2b0a742ec77aedb7ff4dd1ff38e265dffe6918372e4267c384ef0527bd229e04" Jul 23 19:50:07.491283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.491244 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:50:07.678564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678519 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-credential-socket\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678564 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678568 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-socket\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a20541-9cc9-4799-8208-4bad7dc04880-istiod-ca-cert\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678613 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-podinfo\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-token\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678712 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-data\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678773 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-envoy\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-certs\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678833 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5qrg\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-kube-api-access-w5qrg\") pod \"c2a20541-9cc9-4799-8208-4bad7dc04880\" (UID: \"c2a20541-9cc9-4799-8208-4bad7dc04880\") " Jul 23 19:50:07.678878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678864 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:50:07.679280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.678873 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:50:07.679280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.679120 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a20541-9cc9-4799-8208-4bad7dc04880-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:50:07.679280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.679161 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:50:07.679280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.679247 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a20541-9cc9-4799-8208-4bad7dc04880-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.679280 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.679267 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.679477 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.679283 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.679649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.679625 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:50:07.681245 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.681221 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:50:07.681437 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.681418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:50:07.681633 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.681606 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:50:07.681698 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.681609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-kube-api-access-w5qrg" (OuterVolumeSpecName: "kube-api-access-w5qrg") pod "c2a20541-9cc9-4799-8208-4bad7dc04880" (UID: "c2a20541-9cc9-4799-8208-4bad7dc04880"). InnerVolumeSpecName "kube-api-access-w5qrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:50:07.780429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.780316 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.780429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.780359 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.780429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.780372 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.780429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.780383 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.780429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.780393 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a20541-9cc9-4799-8208-4bad7dc04880-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:07.780429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:07.780404 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5qrg\" (UniqueName: \"kubernetes.io/projected/c2a20541-9cc9-4799-8208-4bad7dc04880-kube-api-access-w5qrg\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:50:08.475585 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:08.475555 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" Jul 23 19:50:08.476004 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:08.475548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4" event={"ID":"c2a20541-9cc9-4799-8208-4bad7dc04880","Type":"ContainerDied","Data":"864b1aa7e536df2cbbd7104b069df2963e301f8791d75ea9b27da98e063d4543"} Jul 23 19:50:08.476004 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:08.475706 2574 scope.go:117] "RemoveContainer" containerID="c6d41a6345f9f45e696fbd8e28bdc6847bbb6e06b4a8bb8666cebc1f01f4497b" Jul 23 19:50:08.510311 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:08.510277 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4"] Jul 23 19:50:08.514759 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:08.514714 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f275417d-openshift-default-78d88694f5-bvrl4"] Jul 23 19:50:08.848114 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:08.848032 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" path="/var/lib/kubelet/pods/c2a20541-9cc9-4799-8208-4bad7dc04880/volumes" Jul 23 19:50:52.950712 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:52.950678 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:50:52.957770 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:52.957724 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:50:56.876940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.876904 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp"] Jul 23 19:50:56.877582 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877560 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" Jul 23 19:50:56.877632 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877589 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" Jul 23 19:50:56.877632 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877610 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ce3d38-5567-4db1-be1d-a9f751df83c5" containerName="main" Jul 23 19:50:56.877632 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877620 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ce3d38-5567-4db1-be1d-a9f751df83c5" containerName="main" Jul 23 19:50:56.877743 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877638 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" Jul 23 19:50:56.877743 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877648 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" Jul 23 19:50:56.877817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877765 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ce3d38-5567-4db1-be1d-a9f751df83c5" containerName="main" Jul 23 19:50:56.877817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877780 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" Jul 23 19:50:56.877817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.877794 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a20541-9cc9-4799-8208-4bad7dc04880" containerName="istio-proxy" Jul 23 19:50:56.881346 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.881328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:56.884069 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.884048 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52a527-openshift-default-dockercfg-ztdtk\"" Jul 23 19:50:56.895439 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:56.895403 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp"] Jul 23 19:50:57.020444 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-data\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4svh\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-kube-api-access-k4svh\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-podinfo\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istiod-ca-cert\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020637 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-token\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020910 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-envoy\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020910 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-certs\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020910 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-socket\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.020910 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.020833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-credential-socket\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.121702 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-podinfo\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.121914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istiod-ca-cert\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.121914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-token\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.121914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-envoy\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.121914 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-certs\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-socket\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-credential-socket\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.121979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-data\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.122013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4svh\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-kube-api-access-k4svh\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.122370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-socket\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.122388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-certs\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122526 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.122446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-credential-socket\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122526 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.122456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-data\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.122668 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.122648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istiod-ca-cert\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.124281 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.124256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-envoy\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.124389 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.124349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-podinfo\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.130612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.130548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-token\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.130612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.130564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4svh\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-kube-api-access-k4svh\") pod \"e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.192625 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.192580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:57.345656 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.345628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp"] Jul 23 19:50:57.348393 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:50:57.348356 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471a339d_9bf5_4563_bbb4_b3c9e0a5c837.slice/crio-8657a5d5526163d4c81e7d08b63316e84a17748a7d1e754d1637a54322012594 WatchSource:0}: Error finding container 8657a5d5526163d4c81e7d08b63316e84a17748a7d1e754d1637a54322012594: Status 404 returned error can't find the container with id 8657a5d5526163d4c81e7d08b63316e84a17748a7d1e754d1637a54322012594 Jul 23 19:50:57.350658 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.350623 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:50:57.350778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.350692 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:50:57.350778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.350722 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:50:57.743790 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.743751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" event={"ID":"471a339d-9bf5-4563-bbb4-b3c9e0a5c837","Type":"ContainerStarted","Data":"cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907"} Jul 23 19:50:57.743978 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.743805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" event={"ID":"471a339d-9bf5-4563-bbb4-b3c9e0a5c837","Type":"ContainerStarted","Data":"8657a5d5526163d4c81e7d08b63316e84a17748a7d1e754d1637a54322012594"} Jul 23 19:50:57.768302 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:57.768225 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podStartSLOduration=1.7682034340000001 podStartE2EDuration="1.768203434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:50:57.766615228 +0000 UTC m=+2105.339235768" watchObservedRunningTime="2026-07-23 19:50:57.768203434 +0000 UTC m=+2105.340823977" Jul 23 19:50:58.193417 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:58.193379 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:58.198392 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:58.198367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:58.749130 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:58.749101 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:50:58.750182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:50:58.750159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:51:04.747261 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.747225 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx"] Jul 23 19:51:04.751511 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.751471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.754254 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.754233 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef2e0a-openshift-default-dockercfg-qspq5\"" Jul 23 19:51:04.763566 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.763535 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx"] Jul 23 19:51:04.793823 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.793785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hjs\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-kube-api-access-p4hjs\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.793823 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.793826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25a49aa-b428-4659-850c-c107a834f568-istiod-ca-cert\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794115 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.793846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25a49aa-b428-4659-850c-c107a834f568-istio-podinfo\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794115 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.793934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-istio-token\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794115 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.793992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-socket\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794115 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.794048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-envoy\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794115 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.794078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-credential-socket\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794398 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.794141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-data\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.794398 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.794160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-certs\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.894889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.894853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-credential-socket\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.894889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.894903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-data\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.894923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-certs\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.894963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hjs\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-kube-api-access-p4hjs\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.894985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25a49aa-b428-4659-850c-c107a834f568-istiod-ca-cert\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25a49aa-b428-4659-850c-c107a834f568-istio-podinfo\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-istio-token\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-socket\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895153 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-envoy\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895371 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-credential-socket\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-data\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895521 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-certs\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895700 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-socket\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.895854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.895832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25a49aa-b428-4659-850c-c107a834f568-istiod-ca-cert\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.897662 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.897639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25a49aa-b428-4659-850c-c107a834f568-istio-podinfo\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.897789 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.897671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-envoy\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.903506 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.903481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-istio-token\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:04.903710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:04.903686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hjs\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-kube-api-access-p4hjs\") pod \"e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:05.063911 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.063825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:05.212919 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.212891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx"] Jul 23 19:51:05.214573 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:51:05.214543 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25a49aa_b428_4659_850c_c107a834f568.slice/crio-59af89f20646e515f7a064544c4ad58c33174381650f2479563de693d75d79f5 WatchSource:0}: Error finding container 59af89f20646e515f7a064544c4ad58c33174381650f2479563de693d75d79f5: Status 404 returned error can't find the container with id 59af89f20646e515f7a064544c4ad58c33174381650f2479563de693d75d79f5 Jul 23 19:51:05.217089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.217048 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:05.217202 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.217151 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:05.217264 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.217227 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:05.787530 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.787493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" event={"ID":"e25a49aa-b428-4659-850c-c107a834f568","Type":"ContainerStarted","Data":"88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5"} Jul 23 19:51:05.787997 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.787580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" event={"ID":"e25a49aa-b428-4659-850c-c107a834f568","Type":"ContainerStarted","Data":"59af89f20646e515f7a064544c4ad58c33174381650f2479563de693d75d79f5"} Jul 23 19:51:05.807125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:05.807076 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podStartSLOduration=1.807060661 podStartE2EDuration="1.807060661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:51:05.805877195 +0000 UTC m=+2113.378497732" watchObservedRunningTime="2026-07-23 19:51:05.807060661 +0000 UTC m=+2113.379681200" Jul 23 19:51:06.064589 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:06.064493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:06.070175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:06.070146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:06.793011 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:06.792980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:06.793952 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:06.793931 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:14.364129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.364090 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx"] Jul 23 19:51:14.370468 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.370440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.375004 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.374649 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52a527\"/\"openshift-service-ca.crt\"" Jul 23 19:51:14.375004 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.374825 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52a527\"/\"kube-root-ca.crt\"" Jul 23 19:51:14.375004 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.374968 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52a527\"/\"default-dockercfg-m2j4c\"" Jul 23 19:51:14.375583 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.374659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52a527\"/\"test-model-52a527-kserve-self-signed-certs\"" Jul 23 19:51:14.378429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.378380 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx"] Jul 23 19:51:14.491401 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.491366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-dshm\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.491401 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.491407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-tmp-dir\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.491644 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.491536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-home\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.491644 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.491575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-model-cache\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.491644 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.491627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tsqg\" (UniqueName: \"kubernetes.io/projected/a55b9602-eedb-4813-a958-31b701e25d63-kube-api-access-6tsqg\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.491804 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.491650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55b9602-eedb-4813-a958-31b701e25d63-tls-certs\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592410 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-dshm\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592410 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-tmp-dir\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-home\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-model-cache\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tsqg\" (UniqueName: \"kubernetes.io/projected/a55b9602-eedb-4813-a958-31b701e25d63-kube-api-access-6tsqg\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592641 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55b9602-eedb-4813-a958-31b701e25d63-tls-certs\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592922 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-tmp-dir\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.592991 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592943 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-home\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.593035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.592998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-model-cache\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.594683 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.594662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-dshm\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.595061 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.595043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55b9602-eedb-4813-a958-31b701e25d63-tls-certs\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.601795 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.601766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tsqg\" (UniqueName: \"kubernetes.io/projected/a55b9602-eedb-4813-a958-31b701e25d63-kube-api-access-6tsqg\") pod \"test-model-52a527-kserve-5944c755d5-btlbx\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.682705 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.682665 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:14.791999 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.791954 2574 patch_prober.go:28] interesting pod/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 19:51:14.792204 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.792047 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 19:51:14.834539 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:14.834513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx"] Jul 23 19:51:14.836799 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:51:14.836768 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55b9602_eedb_4813_a958_31b701e25d63.slice/crio-43d152c09dfadd48502344c82089811603ab39ed06df49dbe49539c2a49049a2 WatchSource:0}: Error finding container 43d152c09dfadd48502344c82089811603ab39ed06df49dbe49539c2a49049a2: Status 404 returned error can't find the container with id 43d152c09dfadd48502344c82089811603ab39ed06df49dbe49539c2a49049a2 Jul 23 19:51:15.845167 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:15.845131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" event={"ID":"a55b9602-eedb-4813-a958-31b701e25d63","Type":"ContainerStarted","Data":"fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06"} Jul 23 19:51:15.847518 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:15.845179 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" event={"ID":"a55b9602-eedb-4813-a958-31b701e25d63","Type":"ContainerStarted","Data":"43d152c09dfadd48502344c82089811603ab39ed06df49dbe49539c2a49049a2"} Jul 23 19:51:15.863766 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:15.863708 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" podStartSLOduration=1.8636922839999999 podStartE2EDuration="1.863692284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:51:15.86187694 +0000 UTC m=+2123.434497486" watchObservedRunningTime="2026-07-23 19:51:15.863692284 +0000 UTC m=+2123.436312824" Jul 23 19:51:16.861126 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:16.861089 2574 generic.go:356] "Generic (PLEG): container finished" podID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerID="cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907" exitCode=0 Jul 23 19:51:16.861644 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:16.861171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" event={"ID":"471a339d-9bf5-4563-bbb4-b3c9e0a5c837","Type":"ContainerDied","Data":"cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907"} Jul 23 19:51:16.861883 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:16.861862 2574 scope.go:117] "RemoveContainer" containerID="cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907" Jul 23 19:51:16.862926 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:16.862896 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:16.863030 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:16.862977 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:16.863030 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:16.863007 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:17.872342 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:17.872302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" event={"ID":"471a339d-9bf5-4563-bbb4-b3c9e0a5c837","Type":"ContainerStarted","Data":"023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8"} Jul 23 19:51:18.193635 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:18.193595 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:51:18.193843 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:18.193804 2574 patch_prober.go:28] interesting pod/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 23 19:51:18.193890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:18.193864 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 23 19:51:19.193745 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:19.193701 2574 patch_prober.go:28] interesting pod/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 23 19:51:19.194138 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:19.193788 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 23 19:51:20.193535 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:20.193370 2574 patch_prober.go:28] interesting pod/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 23 19:51:20.193535 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:20.193430 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 23 19:51:21.193799 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:21.193758 2574 patch_prober.go:28] interesting pod/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 23 19:51:21.194200 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:21.193821 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 23 19:51:22.197890 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:22.197857 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:51:22.198395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:22.198214 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:51:22.199129 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:22.199109 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:51:22.835963 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:22.835906 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 19:51:22.836175 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:22.836020 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 19:51:23.912352 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:23.912316 2574 generic.go:356] "Generic (PLEG): container finished" podID="e25a49aa-b428-4659-850c-c107a834f568" containerID="88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5" exitCode=0 Jul 23 19:51:23.912872 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:23.912389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" event={"ID":"e25a49aa-b428-4659-850c-c107a834f568","Type":"ContainerDied","Data":"88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5"} Jul 23 19:51:23.912872 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:23.912854 2574 scope.go:117] "RemoveContainer" containerID="88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5" Jul 23 19:51:23.913846 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:23.913819 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:23.913904 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:23.913883 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:23.913942 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:23.913919 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:51:24.683638 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:24.683600 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:24.683871 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:24.683681 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:24.696304 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:24.696279 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:24.918796 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:24.918760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" event={"ID":"e25a49aa-b428-4659-850c-c107a834f568","Type":"ContainerStarted","Data":"7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75"} Jul 23 19:51:24.933163 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:24.933128 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:51:25.064628 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:25.064467 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:25.064833 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:25.064807 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 19:51:25.064933 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:25.064870 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 19:51:26.065066 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:26.065035 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 19:51:26.065469 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:26.065089 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 19:51:27.064282 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:27.064243 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 19:51:27.064497 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:27.064307 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 19:51:28.064442 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:28.064405 2574 patch_prober.go:28] interesting pod/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 23 19:51:28.064870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:28.064460 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 23 19:51:29.067555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:29.067522 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:29.067992 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:29.067906 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:51:29.068698 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:51:29.068681 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:52:31.829841 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:31.829807 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx"] Jul 23 19:52:46.850551 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:46.850477 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" podUID="a55b9602-eedb-4813-a958-31b701e25d63" containerName="main" containerID="cri-o://fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06" gracePeriod=45 Jul 23 19:52:47.111694 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.111626 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:52:47.198045 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198010 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-model-cache\") pod \"a55b9602-eedb-4813-a958-31b701e25d63\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " Jul 23 19:52:47.198045 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198052 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55b9602-eedb-4813-a958-31b701e25d63-tls-certs\") pod \"a55b9602-eedb-4813-a958-31b701e25d63\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " Jul 23 19:52:47.198255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198098 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-dshm\") pod \"a55b9602-eedb-4813-a958-31b701e25d63\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " Jul 23 19:52:47.198255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198127 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-home\") pod \"a55b9602-eedb-4813-a958-31b701e25d63\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " Jul 23 19:52:47.198344 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198287 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-tmp-dir\") pod \"a55b9602-eedb-4813-a958-31b701e25d63\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " Jul 23 19:52:47.198344 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198327 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tsqg\" (UniqueName: \"kubernetes.io/projected/a55b9602-eedb-4813-a958-31b701e25d63-kube-api-access-6tsqg\") pod \"a55b9602-eedb-4813-a958-31b701e25d63\" (UID: \"a55b9602-eedb-4813-a958-31b701e25d63\") " Jul 23 19:52:47.198460 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198389 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-model-cache" (OuterVolumeSpecName: "model-cache") pod "a55b9602-eedb-4813-a958-31b701e25d63" (UID: "a55b9602-eedb-4813-a958-31b701e25d63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:47.198578 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198472 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-home" (OuterVolumeSpecName: "home") pod "a55b9602-eedb-4813-a958-31b701e25d63" (UID: "a55b9602-eedb-4813-a958-31b701e25d63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:47.198578 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a55b9602-eedb-4813-a958-31b701e25d63" (UID: "a55b9602-eedb-4813-a958-31b701e25d63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:47.198716 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198696 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-tmp-dir\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:47.198811 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198722 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-model-cache\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:47.198811 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.198768 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-home\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:47.200533 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.200508 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-dshm" (OuterVolumeSpecName: "dshm") pod "a55b9602-eedb-4813-a958-31b701e25d63" (UID: "a55b9602-eedb-4813-a958-31b701e25d63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:47.200664 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.200636 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55b9602-eedb-4813-a958-31b701e25d63-kube-api-access-6tsqg" (OuterVolumeSpecName: "kube-api-access-6tsqg") pod "a55b9602-eedb-4813-a958-31b701e25d63" (UID: "a55b9602-eedb-4813-a958-31b701e25d63"). InnerVolumeSpecName "kube-api-access-6tsqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:52:47.200664 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.200633 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55b9602-eedb-4813-a958-31b701e25d63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a55b9602-eedb-4813-a958-31b701e25d63" (UID: "a55b9602-eedb-4813-a958-31b701e25d63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:52:47.299458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.299418 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tsqg\" (UniqueName: \"kubernetes.io/projected/a55b9602-eedb-4813-a958-31b701e25d63-kube-api-access-6tsqg\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:47.299458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.299450 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55b9602-eedb-4813-a958-31b701e25d63-tls-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:47.299458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.299459 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55b9602-eedb-4813-a958-31b701e25d63-dshm\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:47.356000 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.355965 2574 generic.go:356] "Generic (PLEG): container finished" podID="a55b9602-eedb-4813-a958-31b701e25d63" containerID="fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06" exitCode=0 Jul 23 19:52:47.356190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.356016 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" Jul 23 19:52:47.356190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.356050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" event={"ID":"a55b9602-eedb-4813-a958-31b701e25d63","Type":"ContainerDied","Data":"fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06"} Jul 23 19:52:47.356190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.356093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx" event={"ID":"a55b9602-eedb-4813-a958-31b701e25d63","Type":"ContainerDied","Data":"43d152c09dfadd48502344c82089811603ab39ed06df49dbe49539c2a49049a2"} Jul 23 19:52:47.356190 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.356111 2574 scope.go:117] "RemoveContainer" containerID="fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06" Jul 23 19:52:47.366113 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.366095 2574 scope.go:117] "RemoveContainer" containerID="fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06" Jul 23 19:52:47.366381 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:52:47.366359 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06\": container with ID starting with fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06 not found: ID does not exist" containerID="fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06" Jul 23 19:52:47.366444 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.366395 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06"} err="failed to get container status \"fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06\": rpc error: code = NotFound desc = could not find container \"fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06\": container with ID starting with fd63cceb81046b17339db1e28e9a2adbb587756711fc58add2ce36da4af57f06 not found: ID does not exist" Jul 23 19:52:47.378692 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.378461 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx"] Jul 23 19:52:47.380847 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:47.380829 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-52a527/test-model-52a527-kserve-5944c755d5-btlbx"] Jul 23 19:52:48.848319 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:48.848290 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55b9602-eedb-4813-a958-31b701e25d63" path="/var/lib/kubelet/pods/a55b9602-eedb-4813-a958-31b701e25d63/volumes" Jul 23 19:52:54.495982 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:54.495945 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp"] Jul 23 19:52:54.496426 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:54.496211 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" containerID="cri-o://023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8" gracePeriod=30 Jul 23 19:52:59.765180 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.765110 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:52:59.925759 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925704 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-socket\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.925759 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-credential-socket\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925805 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4svh\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-kube-api-access-k4svh\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-data\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925885 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-token\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925901 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-certs\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925927 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-podinfo\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925963 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istiod-ca-cert\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926001 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925993 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-envoy\") pod \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\" (UID: \"471a339d-9bf5-4563-bbb4-b3c9e0a5c837\") " Jul 23 19:52:59.926367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.925998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:59.926367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.926087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:59.926367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.926192 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:59.926367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.926262 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:59.926367 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.926282 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:52:59.926626 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.926387 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:52:59.927061 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.927031 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-data" (OuterVolumeSpecName: "istio-data") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:59.928252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.928214 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:52:59.928252 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.928224 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:52:59.928561 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.928539 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-token" (OuterVolumeSpecName: "istio-token") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:52:59.928709 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:52:59.928689 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-kube-api-access-k4svh" (OuterVolumeSpecName: "kube-api-access-k4svh") pod "471a339d-9bf5-4563-bbb4-b3c9e0a5c837" (UID: "471a339d-9bf5-4563-bbb4-b3c9e0a5c837"). InnerVolumeSpecName "kube-api-access-k4svh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027268 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027296 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027305 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027314 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027321 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027329 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.027351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.027338 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4svh\" (UniqueName: \"kubernetes.io/projected/471a339d-9bf5-4563-bbb4-b3c9e0a5c837-kube-api-access-k4svh\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:00.424167 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.424133 2574 generic.go:356] "Generic (PLEG): container finished" podID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerID="023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8" exitCode=0 Jul 23 19:53:00.424350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.424172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" event={"ID":"471a339d-9bf5-4563-bbb4-b3c9e0a5c837","Type":"ContainerDied","Data":"023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8"} Jul 23 19:53:00.424350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.424184 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" Jul 23 19:53:00.424350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.424217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp" event={"ID":"471a339d-9bf5-4563-bbb4-b3c9e0a5c837","Type":"ContainerDied","Data":"8657a5d5526163d4c81e7d08b63316e84a17748a7d1e754d1637a54322012594"} Jul 23 19:53:00.424350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.424247 2574 scope.go:117] "RemoveContainer" containerID="023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8" Jul 23 19:53:00.433898 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.433874 2574 scope.go:117] "RemoveContainer" containerID="cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907" Jul 23 19:53:00.442207 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.442187 2574 scope.go:117] "RemoveContainer" containerID="023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8" Jul 23 19:53:00.442453 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:53:00.442431 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8\": container with ID starting with 023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8 not found: ID does not exist" containerID="023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8" Jul 23 19:53:00.442506 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.442461 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8"} err="failed to get container status \"023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8\": rpc error: code = NotFound desc = could not find container \"023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8\": container with ID starting with 023c86dccf0ee3ede90bf5b07b8f75b7d16db90e98b98202464457f71a6395f8 not found: ID does not exist" Jul 23 19:53:00.442506 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.442478 2574 scope.go:117] "RemoveContainer" containerID="cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907" Jul 23 19:53:00.442704 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:53:00.442686 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907\": container with ID starting with cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907 not found: ID does not exist" containerID="cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907" Jul 23 19:53:00.442790 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.442714 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907"} err="failed to get container status \"cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907\": rpc error: code = NotFound desc = could not find container \"cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907\": container with ID starting with cf7697ebcf76e0a9621bd147d5bc2db35f85143ab267cdab39364e80d5a08907 not found: ID does not exist" Jul 23 19:53:00.447947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.447924 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp"] Jul 23 19:53:00.455067 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.455044 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52a527-openshift-default-76f678b449-rzpdp"] Jul 23 19:53:00.846916 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:00.846839 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" path="/var/lib/kubelet/pods/471a339d-9bf5-4563-bbb4-b3c9e0a5c837/volumes" Jul 23 19:53:16.266017 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:16.265927 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx"] Jul 23 19:53:16.266501 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:16.266146 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" containerID="cri-o://7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75" gracePeriod=30 Jul 23 19:53:17.111244 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111210 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8"] Jul 23 19:53:17.111631 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111617 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" Jul 23 19:53:17.111631 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111631 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" Jul 23 19:53:17.111822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111642 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55b9602-eedb-4813-a958-31b701e25d63" containerName="main" Jul 23 19:53:17.111822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111648 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55b9602-eedb-4813-a958-31b701e25d63" containerName="main" Jul 23 19:53:17.111822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111659 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" Jul 23 19:53:17.111822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111665 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" Jul 23 19:53:17.111822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111750 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" Jul 23 19:53:17.111822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.111760 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55b9602-eedb-4813-a958-31b701e25d63" containerName="main" Jul 23 19:53:17.116395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.116371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.119166 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.119141 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-081c5e-openshift-default-dockercfg-l7l4t\"" Jul 23 19:53:17.128404 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.128382 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8"] Jul 23 19:53:17.188030 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.187992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fb6552-3d19-4ac5-b8ee-85611783cc23-istiod-ca-cert\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188204 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-credential-socket\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188204 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-podinfo\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188204 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-socket\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188204 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-certs\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-data\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrqh\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-kube-api-access-mlrqh\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-envoy\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.188351 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.188323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-token\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289208 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fb6552-3d19-4ac5-b8ee-85611783cc23-istiod-ca-cert\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-credential-socket\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-podinfo\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-socket\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-certs\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-data\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrqh\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-kube-api-access-mlrqh\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-envoy\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.289649 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-token\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.290087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-data\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.290087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289805 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-socket\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.290087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-credential-socket\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.290087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.289992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-certs\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.290087 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.290002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fb6552-3d19-4ac5-b8ee-85611783cc23-istiod-ca-cert\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.291849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.291824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-envoy\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.292053 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.292036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-podinfo\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.297612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.297587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-token\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.298394 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.298364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrqh\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-kube-api-access-mlrqh\") pod \"e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.427138 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.427096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:17.575887 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.575860 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8"] Jul 23 19:53:17.578292 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:53:17.578262 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fb6552_3d19_4ac5_b8ee_85611783cc23.slice/crio-61bfa51d32c9896fe566076949630356a95e7c136330909ae883e74dd5c8b55e WatchSource:0}: Error finding container 61bfa51d32c9896fe566076949630356a95e7c136330909ae883e74dd5c8b55e: Status 404 returned error can't find the container with id 61bfa51d32c9896fe566076949630356a95e7c136330909ae883e74dd5c8b55e Jul 23 19:53:17.580808 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.580765 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:17.580896 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.580868 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:17.580934 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:17.580911 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:18.526845 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:18.526811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" event={"ID":"40fb6552-3d19-4ac5-b8ee-85611783cc23","Type":"ContainerStarted","Data":"32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672"} Jul 23 19:53:18.527243 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:18.526862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" event={"ID":"40fb6552-3d19-4ac5-b8ee-85611783cc23","Type":"ContainerStarted","Data":"61bfa51d32c9896fe566076949630356a95e7c136330909ae883e74dd5c8b55e"} Jul 23 19:53:18.547547 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:18.547501 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podStartSLOduration=1.547484142 podStartE2EDuration="1.547484142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:53:18.545825097 +0000 UTC m=+2246.118445638" watchObservedRunningTime="2026-07-23 19:53:18.547484142 +0000 UTC m=+2246.120104682" Jul 23 19:53:19.427958 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:19.427915 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:19.433480 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:19.433450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:19.533268 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:19.533234 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:19.534350 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:19.534329 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:21.535219 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.535135 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:53:21.548243 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.548199 2574 generic.go:356] "Generic (PLEG): container finished" podID="e25a49aa-b428-4659-850c-c107a834f568" containerID="7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75" exitCode=0 Jul 23 19:53:21.548396 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.548258 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" Jul 23 19:53:21.548396 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.548288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" event={"ID":"e25a49aa-b428-4659-850c-c107a834f568","Type":"ContainerDied","Data":"7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75"} Jul 23 19:53:21.548396 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.548343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx" event={"ID":"e25a49aa-b428-4659-850c-c107a834f568","Type":"ContainerDied","Data":"59af89f20646e515f7a064544c4ad58c33174381650f2479563de693d75d79f5"} Jul 23 19:53:21.548396 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.548375 2574 scope.go:117] "RemoveContainer" containerID="7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75" Jul 23 19:53:21.557486 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.557462 2574 scope.go:117] "RemoveContainer" containerID="88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5" Jul 23 19:53:21.569514 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.569489 2574 scope.go:117] "RemoveContainer" containerID="7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75" Jul 23 19:53:21.570340 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:53:21.570309 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75\": container with ID starting with 7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75 not found: ID does not exist" containerID="7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75" Jul 23 19:53:21.570416 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.570355 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75"} err="failed to get container status \"7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75\": rpc error: code = NotFound desc = could not find container \"7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75\": container with ID starting with 7f58965033df8649fa2e094679e8342988dc38047d20d9043ba252f69070fd75 not found: ID does not exist" Jul 23 19:53:21.570416 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.570381 2574 scope.go:117] "RemoveContainer" containerID="88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5" Jul 23 19:53:21.570770 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:53:21.570725 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5\": container with ID starting with 88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5 not found: ID does not exist" containerID="88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5" Jul 23 19:53:21.570858 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.570779 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5"} err="failed to get container status \"88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5\": rpc error: code = NotFound desc = could not find container \"88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5\": container with ID starting with 88354de15a253d1dffb2033c9801935ea27700a49f0242b14bfb2386d5f927b5 not found: ID does not exist" Jul 23 19:53:21.634250 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25a49aa-b428-4659-850c-c107a834f568-istio-podinfo\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634432 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634278 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4hjs\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-kube-api-access-p4hjs\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634432 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634392 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-socket\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634437 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-istio-token\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634477 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-data\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634516 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-certs\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634615 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634590 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-credential-socket\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634903 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634626 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-envoy\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634903 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634683 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:53:21.634903 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634699 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25a49aa-b428-4659-850c-c107a834f568-istiod-ca-cert\") pod \"e25a49aa-b428-4659-850c-c107a834f568\" (UID: \"e25a49aa-b428-4659-850c-c107a834f568\") " Jul 23 19:53:21.634903 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634866 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:53:21.634903 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.634879 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:53:21.635187 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.635163 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.635251 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.635189 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.635251 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.635192 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25a49aa-b428-4659-850c-c107a834f568-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:53:21.635251 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.635203 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.636191 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.636164 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-data" (OuterVolumeSpecName: "istio-data") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:53:21.636980 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.636958 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-istio-token" (OuterVolumeSpecName: "istio-token") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:53:21.637065 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.636978 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-kube-api-access-p4hjs" (OuterVolumeSpecName: "kube-api-access-p4hjs") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "kube-api-access-p4hjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:53:21.637438 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.637420 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e25a49aa-b428-4659-850c-c107a834f568-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:53:21.637519 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.637499 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e25a49aa-b428-4659-850c-c107a834f568" (UID: "e25a49aa-b428-4659-850c-c107a834f568"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:53:21.736830 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.736788 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25a49aa-b428-4659-850c-c107a834f568-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.736830 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.736827 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25a49aa-b428-4659-850c-c107a834f568-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.737054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.736843 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4hjs\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-kube-api-access-p4hjs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.737054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.736857 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25a49aa-b428-4659-850c-c107a834f568-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.737054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.736870 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.737054 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.736882 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25a49aa-b428-4659-850c-c107a834f568-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:53:21.881923 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.881887 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx"] Jul 23 19:53:21.887940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:21.887908 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef2e0a-openshift-default-76c488c57c-gkdfx"] Jul 23 19:53:22.848542 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:22.848503 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e25a49aa-b428-4659-850c-c107a834f568" path="/var/lib/kubelet/pods/e25a49aa-b428-4659-850c-c107a834f568/volumes" Jul 23 19:53:25.667348 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.667308 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69"] Jul 23 19:53:25.667982 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.667962 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" Jul 23 19:53:25.668063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.667986 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" Jul 23 19:53:25.668063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.668023 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" Jul 23 19:53:25.668063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.668033 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" Jul 23 19:53:25.668227 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.668134 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" Jul 23 19:53:25.668227 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.668147 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25a49aa-b428-4659-850c-c107a834f568" containerName="istio-proxy" Jul 23 19:53:25.668227 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.668157 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="471a339d-9bf5-4563-bbb4-b3c9e0a5c837" containerName="istio-proxy" Jul 23 19:53:25.672871 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.672834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.675676 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.675652 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dbc659-openshift-default-dockercfg-s2sjw\"" Jul 23 19:53:25.684237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.684199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69"] Jul 23 19:53:25.776032 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-data\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776032 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-token\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776273 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccsv\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-kube-api-access-7ccsv\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776273 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istiod-ca-cert\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776273 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776203 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-podinfo\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776273 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-certs\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776287 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-socket\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-credential-socket\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.776435 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.776325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-envoy\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877303 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccsv\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-kube-api-access-7ccsv\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istiod-ca-cert\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-podinfo\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-certs\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-socket\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-credential-socket\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877532 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-envoy\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-data\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.877849 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.877624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-token\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.878057 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.878032 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-credential-socket\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.878142 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.878057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-certs\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.878142 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.878127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-data\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.878251 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.878148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-socket\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.878522 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.878502 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istiod-ca-cert\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.879874 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.879847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-envoy\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.880064 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.880048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-podinfo\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.886679 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.886654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-token\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.887029 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.887006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccsv\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-kube-api-access-7ccsv\") pod \"e2e-api-b-dbc659-openshift-default-57987bc867-pvt69\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:25.984378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:25.984282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:26.141645 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.141619 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69"] Jul 23 19:53:26.154103 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.153755 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:26.154103 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.153881 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:26.154103 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.153981 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:26.579429 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.579388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" event={"ID":"645e235a-63bc-4f6c-9c05-a7d8aba03a04","Type":"ContainerStarted","Data":"4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12"} Jul 23 19:53:26.579648 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.579476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" event={"ID":"645e235a-63bc-4f6c-9c05-a7d8aba03a04","Type":"ContainerStarted","Data":"f81f3f02c6d7b22890f713aa7270df910ac3083e54e819961425c18b790d72a8"} Jul 23 19:53:26.599568 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.599518 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" podStartSLOduration=1.5995016340000001 podStartE2EDuration="1.599501634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:53:26.59764822 +0000 UTC m=+2254.170268761" watchObservedRunningTime="2026-07-23 19:53:26.599501634 +0000 UTC m=+2254.172122175" Jul 23 19:53:26.984457 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:26.984416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:27.989898 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:27.989871 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:28.593407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:28.593379 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:28.594660 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:28.594638 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:33.620401 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:33.620303 2574 generic.go:356] "Generic (PLEG): container finished" podID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerID="32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672" exitCode=0 Jul 23 19:53:33.620401 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:33.620379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" event={"ID":"40fb6552-3d19-4ac5-b8ee-85611783cc23","Type":"ContainerDied","Data":"32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672"} Jul 23 19:53:33.620861 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:33.620845 2574 scope.go:117] "RemoveContainer" containerID="32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672" Jul 23 19:53:33.624390 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:33.624269 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:33.624390 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:33.624355 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:33.624518 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:33.624394 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:34.533458 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:34.533420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:34.627984 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:34.627948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" event={"ID":"40fb6552-3d19-4ac5-b8ee-85611783cc23","Type":"ContainerStarted","Data":"620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d"} Jul 23 19:53:35.427868 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:35.427830 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:35.428979 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:35.428943 2574 patch_prober.go:28] interesting pod/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 19:53:35.429093 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:35.429017 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 19:53:36.428066 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:36.428029 2574 patch_prober.go:28] interesting pod/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 19:53:36.428493 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:36.428083 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 19:53:37.428462 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:37.428428 2574 patch_prober.go:28] interesting pod/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 19:53:37.428938 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:37.428483 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 19:53:38.428470 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:38.428419 2574 patch_prober.go:28] interesting pod/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 23 19:53:38.428923 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:38.428540 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 23 19:53:39.432722 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:39.432692 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:39.433267 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:39.433242 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:39.434263 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:39.434232 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:53:41.681206 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:41.681175 2574 generic.go:356] "Generic (PLEG): container finished" podID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerID="4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12" exitCode=0 Jul 23 19:53:41.681647 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:41.681254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" event={"ID":"645e235a-63bc-4f6c-9c05-a7d8aba03a04","Type":"ContainerDied","Data":"4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12"} Jul 23 19:53:41.681714 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:41.681700 2574 scope.go:117] "RemoveContainer" containerID="4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12" Jul 23 19:53:41.682974 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:41.682934 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:41.683049 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:41.683033 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:41.683099 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:41.683084 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:53:42.689275 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:42.689239 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" event={"ID":"645e235a-63bc-4f6c-9c05-a7d8aba03a04","Type":"ContainerStarted","Data":"ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27"} Jul 23 19:53:42.985508 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:42.985409 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:42.986778 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:42.986746 2574 patch_prober.go:28] interesting pod/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 23 19:53:42.986990 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:42.986960 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 23 19:53:43.593453 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:43.593421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:43.984683 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:43.984646 2574 patch_prober.go:28] interesting pod/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 23 19:53:43.985162 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:43.984712 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 23 19:53:44.984699 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:44.984666 2574 patch_prober.go:28] interesting pod/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 23 19:53:44.985111 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:44.984718 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 23 19:53:45.985586 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:45.985544 2574 patch_prober.go:28] interesting pod/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 23 19:53:45.986115 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:45.985631 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 23 19:53:46.989116 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:46.989085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:53:46.990021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:53:46.989998 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:54:07.463492 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:07.463459 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8"] Jul 23 19:54:07.464000 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:07.463722 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" containerID="cri-o://620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d" gracePeriod=30 Jul 23 19:54:09.435697 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:09.435652 2574 patch_prober.go:28] interesting pod/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 19:54:09.436179 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:09.435726 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 19:54:12.728444 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.728369 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:54:12.847219 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.847188 2574 generic.go:356] "Generic (PLEG): container finished" podID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerID="620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d" exitCode=0 Jul 23 19:54:12.847423 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.847312 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" Jul 23 19:54:12.850144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.850119 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" event={"ID":"40fb6552-3d19-4ac5-b8ee-85611783cc23","Type":"ContainerDied","Data":"620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d"} Jul 23 19:54:12.850264 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.850167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8" event={"ID":"40fb6552-3d19-4ac5-b8ee-85611783cc23","Type":"ContainerDied","Data":"61bfa51d32c9896fe566076949630356a95e7c136330909ae883e74dd5c8b55e"} Jul 23 19:54:12.850264 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.850189 2574 scope.go:117] "RemoveContainer" containerID="620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d" Jul 23 19:54:12.852476 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852454 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-credential-socket\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852521 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-envoy\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852555 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852545 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-certs\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852580 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-data\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852610 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-token\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852659 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852649 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-podinfo\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852830 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852674 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-socket\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.852830 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852778 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:12.852937 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852892 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:12.852937 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852902 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlrqh\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-kube-api-access-mlrqh\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.853045 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.852968 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fb6552-3d19-4ac5-b8ee-85611783cc23-istiod-ca-cert\") pod \"40fb6552-3d19-4ac5-b8ee-85611783cc23\" (UID: \"40fb6552-3d19-4ac5-b8ee-85611783cc23\") " Jul 23 19:54:12.853045 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.853011 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:12.853363 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.853301 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.853363 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.853324 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.853363 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.853338 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.853575 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.853390 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fb6552-3d19-4ac5-b8ee-85611783cc23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:54:12.853639 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.853604 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-data" (OuterVolumeSpecName: "istio-data") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:12.854954 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.854928 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:12.855117 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.855093 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-token" (OuterVolumeSpecName: "istio-token") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:54:12.855378 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.855351 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:54:12.855508 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.855388 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-kube-api-access-mlrqh" (OuterVolumeSpecName: "kube-api-access-mlrqh") pod "40fb6552-3d19-4ac5-b8ee-85611783cc23" (UID: "40fb6552-3d19-4ac5-b8ee-85611783cc23"). InnerVolumeSpecName "kube-api-access-mlrqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:54:12.860064 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.860040 2574 scope.go:117] "RemoveContainer" containerID="32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672" Jul 23 19:54:12.869867 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.869848 2574 scope.go:117] "RemoveContainer" containerID="620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d" Jul 23 19:54:12.870166 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:54:12.870137 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d\": container with ID starting with 620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d not found: ID does not exist" containerID="620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d" Jul 23 19:54:12.870231 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.870177 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d"} err="failed to get container status \"620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d\": rpc error: code = NotFound desc = could not find container \"620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d\": container with ID starting with 620635cb24ff86e25a24379f2b6ac49bd632d154f5a9503e9fcf9e6833d85d7d not found: ID does not exist" Jul 23 19:54:12.870231 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.870203 2574 scope.go:117] "RemoveContainer" containerID="32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672" Jul 23 19:54:12.870491 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:54:12.870472 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672\": container with ID starting with 32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672 not found: ID does not exist" containerID="32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672" Jul 23 19:54:12.870552 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.870501 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672"} err="failed to get container status \"32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672\": rpc error: code = NotFound desc = could not find container \"32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672\": container with ID starting with 32f9e742ae19035363e7c8469a8a855a3032628391835bda4a407a27d6bad672 not found: ID does not exist" Jul 23 19:54:12.954690 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.954652 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.954690 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.954682 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.954690 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.954691 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.954690 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.954700 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fb6552-3d19-4ac5-b8ee-85611783cc23-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.954985 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.954710 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrqh\" (UniqueName: \"kubernetes.io/projected/40fb6552-3d19-4ac5-b8ee-85611783cc23-kube-api-access-mlrqh\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:12.954985 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:12.954719 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fb6552-3d19-4ac5-b8ee-85611783cc23-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:13.169395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:13.169353 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8"] Jul 23 19:54:13.173276 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:13.173241 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-081c5e-openshift-default-5cc57d448c-4h9p8"] Jul 23 19:54:14.847679 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:14.847643 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" path="/var/lib/kubelet/pods/40fb6552-3d19-4ac5-b8ee-85611783cc23/volumes" Jul 23 19:54:34.472785 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:34.472748 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69"] Jul 23 19:54:34.473221 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:34.472970 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" containerID="cri-o://ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27" gracePeriod=30 Jul 23 19:54:35.425137 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.425099 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d"] Jul 23 19:54:35.425817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.425790 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" Jul 23 19:54:35.425817 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.425818 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" Jul 23 19:54:35.426002 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.425946 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" Jul 23 19:54:35.426002 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.425962 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" Jul 23 19:54:35.429270 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.429250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.431973 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.431954 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-459a5a-openshift-default-dockercfg-5fsvl\"" Jul 23 19:54:35.443477 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.443448 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d"] Jul 23 19:54:35.461010 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.460980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-certs\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461181 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-token\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461181 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-credential-socket\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461181 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istiod-ca-cert\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461181 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-envoy\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461181 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfzm6\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-kube-api-access-mfzm6\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-podinfo\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-data\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.461395 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.461306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-socket\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.562623 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-certs\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-token\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-credential-socket\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istiod-ca-cert\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-envoy\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfzm6\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-kube-api-access-mfzm6\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-podinfo\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-data\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.562883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-socket\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563089 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.563080 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-credential-socket\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563473 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.563160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-certs\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563473 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.563194 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-socket\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563473 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.563343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-data\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.563565 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.563491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istiod-ca-cert\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.565053 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.565028 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-envoy\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.565346 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.565330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-podinfo\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.570947 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.570924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-token\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.571037 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.570963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfzm6\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-kube-api-access-mfzm6\") pod \"e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.739636 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.739540 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:35.883297 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.883273 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d"] Jul 23 19:54:35.884904 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:54:35.884877 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466f3f89_f60e_4c11_9900_f58d3f0a7d66.slice/crio-a9fe377e62ceec424f74e56d6fd7d1504be5da280b7dd0fee4347f5de3c9d162 WatchSource:0}: Error finding container a9fe377e62ceec424f74e56d6fd7d1504be5da280b7dd0fee4347f5de3c9d162: Status 404 returned error can't find the container with id a9fe377e62ceec424f74e56d6fd7d1504be5da280b7dd0fee4347f5de3c9d162 Jul 23 19:54:35.887610 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.887577 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:35.887685 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.887645 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:35.887685 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.887674 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:35.979186 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.979146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" event={"ID":"466f3f89-f60e-4c11-9900-f58d3f0a7d66","Type":"ContainerStarted","Data":"684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890"} Jul 23 19:54:35.979369 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.979247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" event={"ID":"466f3f89-f60e-4c11-9900-f58d3f0a7d66","Type":"ContainerStarted","Data":"a9fe377e62ceec424f74e56d6fd7d1504be5da280b7dd0fee4347f5de3c9d162"} Jul 23 19:54:35.998643 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:35.998535 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podStartSLOduration=0.998520299 podStartE2EDuration="998.520299ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:54:35.99781559 +0000 UTC m=+2323.570436135" watchObservedRunningTime="2026-07-23 19:54:35.998520299 +0000 UTC m=+2323.571140838" Jul 23 19:54:36.739983 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:36.739942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:36.745144 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:36.745112 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:36.984498 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:36.984465 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:36.985450 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:36.985432 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:39.745919 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.745860 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:54:39.811723 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.811686 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-token\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.811936 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.811776 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-certs\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.811936 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.811849 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-credential-socket\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.811936 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.811882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ccsv\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-kube-api-access-7ccsv\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.811936 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.811905 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istiod-ca-cert\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.811936 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.811924 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-envoy\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.812191 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-podinfo\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.812191 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812120 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:39.812191 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812129 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-data\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.812365 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:39.812365 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812202 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-socket\") pod \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\" (UID: \"645e235a-63bc-4f6c-9c05-a7d8aba03a04\") " Jul 23 19:54:39.812365 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812303 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 19:54:39.812550 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812532 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.812613 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812555 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.812613 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812569 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.812613 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.812598 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:39.813143 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.813117 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-data" (OuterVolumeSpecName: "istio-data") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:39.814218 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.814185 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-kube-api-access-7ccsv" (OuterVolumeSpecName: "kube-api-access-7ccsv") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "kube-api-access-7ccsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:54:39.814328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.814255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:54:39.814475 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.814450 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 19:54:39.814576 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.814470 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-token" (OuterVolumeSpecName: "istio-token") pod "645e235a-63bc-4f6c-9c05-a7d8aba03a04" (UID: "645e235a-63bc-4f6c-9c05-a7d8aba03a04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:54:39.913995 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.913960 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.913995 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.913987 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.913995 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.913998 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.913995 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.914006 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.914275 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.914016 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ccsv\" (UniqueName: \"kubernetes.io/projected/645e235a-63bc-4f6c-9c05-a7d8aba03a04-kube-api-access-7ccsv\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:39.914275 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:39.914025 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645e235a-63bc-4f6c-9c05-a7d8aba03a04-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:54:40.001612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.001517 2574 generic.go:356] "Generic (PLEG): container finished" podID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerID="ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27" exitCode=0 Jul 23 19:54:40.001612 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.001566 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" Jul 23 19:54:40.001878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.001606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" event={"ID":"645e235a-63bc-4f6c-9c05-a7d8aba03a04","Type":"ContainerDied","Data":"ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27"} Jul 23 19:54:40.001878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.001670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69" event={"ID":"645e235a-63bc-4f6c-9c05-a7d8aba03a04","Type":"ContainerDied","Data":"f81f3f02c6d7b22890f713aa7270df910ac3083e54e819961425c18b790d72a8"} Jul 23 19:54:40.001878 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.001703 2574 scope.go:117] "RemoveContainer" containerID="ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27" Jul 23 19:54:40.011078 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.011047 2574 scope.go:117] "RemoveContainer" containerID="4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12" Jul 23 19:54:40.019086 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.019064 2574 scope.go:117] "RemoveContainer" containerID="ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27" Jul 23 19:54:40.019341 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:54:40.019322 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27\": container with ID starting with ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27 not found: ID does not exist" containerID="ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27" Jul 23 19:54:40.019402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.019350 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27"} err="failed to get container status \"ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27\": rpc error: code = NotFound desc = could not find container \"ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27\": container with ID starting with ec07f521a7fd2afaf33924ba971f46d5c494296fbe669492b5f7d7f3bbdb4e27 not found: ID does not exist" Jul 23 19:54:40.019402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.019368 2574 scope.go:117] "RemoveContainer" containerID="4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12" Jul 23 19:54:40.019666 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:54:40.019646 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12\": container with ID starting with 4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12 not found: ID does not exist" containerID="4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12" Jul 23 19:54:40.019761 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.019673 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12"} err="failed to get container status \"4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12\": rpc error: code = NotFound desc = could not find container \"4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12\": container with ID starting with 4eda11c73a688e0f54f166cf833543636f4186b775df6c85b8651debd7555c12 not found: ID does not exist" Jul 23 19:54:40.023299 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.023280 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69"] Jul 23 19:54:40.025167 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.025148 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dbc659-openshift-default-57987bc867-pvt69"] Jul 23 19:54:40.853980 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:40.853941 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" path="/var/lib/kubelet/pods/645e235a-63bc-4f6c-9c05-a7d8aba03a04/volumes" Jul 23 19:54:43.081542 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.081508 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5"] Jul 23 19:54:43.082100 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082080 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" Jul 23 19:54:43.082182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082103 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fb6552-3d19-4ac5-b8ee-85611783cc23" containerName="istio-proxy" Jul 23 19:54:43.082182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082144 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" Jul 23 19:54:43.082182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082153 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" Jul 23 19:54:43.082182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082171 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" Jul 23 19:54:43.082182 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082179 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" Jul 23 19:54:43.082431 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082272 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" Jul 23 19:54:43.082431 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.082286 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="645e235a-63bc-4f6c-9c05-a7d8aba03a04" containerName="istio-proxy" Jul 23 19:54:43.086799 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.086774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.090045 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.090021 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-829079-openshift-default-dockercfg-8zmp5\"" Jul 23 19:54:43.101527 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.101499 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5"] Jul 23 19:54:43.146031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.145988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcld\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-kube-api-access-rrcld\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146031 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-socket\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3ba37c-da24-4343-ba93-548952445a34-istio-podinfo\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-credential-socket\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-istio-token\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3ba37c-da24-4343-ba93-548952445a34-istiod-ca-cert\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-certs\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146294 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-envoy\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.146518 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.146355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-data\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.246872 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.246835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-data\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.246889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcld\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-kube-api-access-rrcld\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.246915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-socket\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.246944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3ba37c-da24-4343-ba93-548952445a34-istio-podinfo\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.246964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-credential-socket\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247060 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-istio-token\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247322 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3ba37c-da24-4343-ba93-548952445a34-istiod-ca-cert\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247322 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-certs\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247322 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-envoy\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247322 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-data\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247539 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-socket\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247539 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247469 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-credential-socket\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247660 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-certs\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.247870 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.247847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3ba37c-da24-4343-ba93-548952445a34-istiod-ca-cert\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.249785 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.249766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-envoy\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.250020 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.249997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3ba37c-da24-4343-ba93-548952445a34-istio-podinfo\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.255245 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.255221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-istio-token\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.255491 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.255475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcld\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-kube-api-access-rrcld\") pod \"e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.397469 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.397376 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:43.547283 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.547249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5"] Jul 23 19:54:43.548455 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:54:43.548429 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3ba37c_da24_4343_ba93_548952445a34.slice/crio-9ffefe05e0949cfd267660b587bd971365a1fba123942a4ae891f7df791e5e90 WatchSource:0}: Error finding container 9ffefe05e0949cfd267660b587bd971365a1fba123942a4ae891f7df791e5e90: Status 404 returned error can't find the container with id 9ffefe05e0949cfd267660b587bd971365a1fba123942a4ae891f7df791e5e90 Jul 23 19:54:43.550741 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.550694 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:43.550821 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.550786 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:43.550821 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:43.550814 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:44.024943 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:44.024909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" event={"ID":"cf3ba37c-da24-4343-ba93-548952445a34","Type":"ContainerStarted","Data":"c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a"} Jul 23 19:54:44.025151 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:44.024961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" event={"ID":"cf3ba37c-da24-4343-ba93-548952445a34","Type":"ContainerStarted","Data":"9ffefe05e0949cfd267660b587bd971365a1fba123942a4ae891f7df791e5e90"} Jul 23 19:54:44.047782 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:44.047713 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podStartSLOduration=1.047694703 podStartE2EDuration="1.047694703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:54:44.046753931 +0000 UTC m=+2331.619374462" watchObservedRunningTime="2026-07-23 19:54:44.047694703 +0000 UTC m=+2331.620315243" Jul 23 19:54:44.398408 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:44.398291 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:44.403583 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:44.403557 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:45.031366 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:45.031336 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:45.032254 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:45.032223 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:54:53.025976 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:53.025928 2574 patch_prober.go:28] interesting pod/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 19:54:53.026455 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:53.026023 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 19:54:55.094624 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:55.094589 2574 generic.go:356] "Generic (PLEG): container finished" podID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerID="684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890" exitCode=0 Jul 23 19:54:55.095053 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:55.094635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" event={"ID":"466f3f89-f60e-4c11-9900-f58d3f0a7d66","Type":"ContainerDied","Data":"684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890"} Jul 23 19:54:55.095098 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:55.095058 2574 scope.go:117] "RemoveContainer" containerID="684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890" Jul 23 19:54:55.096147 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:55.096114 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:55.096237 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:55.096214 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:55.096276 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:55.096260 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:54:56.101688 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:56.101649 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" event={"ID":"466f3f89-f60e-4c11-9900-f58d3f0a7d66","Type":"ContainerStarted","Data":"8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0"} Jul 23 19:54:56.740302 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:56.740259 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:54:56.740720 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:56.740693 2574 patch_prober.go:28] interesting pod/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 19:54:56.740940 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:56.740911 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 19:54:57.740081 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:57.740041 2574 patch_prober.go:28] interesting pod/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 19:54:57.741149 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:57.740684 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 19:54:58.740833 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:58.740780 2574 patch_prober.go:28] interesting pod/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 19:54:58.741356 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:58.740875 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 19:54:59.740630 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:59.740586 2574 patch_prober.go:28] interesting pod/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 23 19:54:59.740977 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:54:59.740944 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 23 19:55:00.744546 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:00.744513 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:55:00.745048 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:00.744805 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:55:00.745578 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:00.745560 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 19:55:01.073021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:01.072927 2574 patch_prober.go:28] interesting pod/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 19:55:01.073021 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:01.073004 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 19:55:03.145217 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:03.145128 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf3ba37c-da24-4343-ba93-548952445a34" containerID="c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a" exitCode=0 Jul 23 19:55:03.145614 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:03.145207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" event={"ID":"cf3ba37c-da24-4343-ba93-548952445a34","Type":"ContainerDied","Data":"c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a"} Jul 23 19:55:03.145816 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:03.145798 2574 scope.go:117] "RemoveContainer" containerID="c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a" Jul 23 19:55:03.146972 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:03.146943 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:55:03.147082 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:03.147029 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:55:03.147082 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:03.147055 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 23 19:55:04.159119 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:04.159079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" event={"ID":"cf3ba37c-da24-4343-ba93-548952445a34","Type":"ContainerStarted","Data":"70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91"} Jul 23 19:55:04.398209 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:04.398163 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:55:04.398445 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:04.398421 2574 patch_prober.go:28] interesting pod/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 19:55:04.398536 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:04.398513 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 19:55:05.398549 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:05.398513 2574 patch_prober.go:28] interesting pod/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 19:55:05.398986 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:05.398568 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 19:55:06.398063 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:06.398029 2574 patch_prober.go:28] interesting pod/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 19:55:06.398249 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:06.398083 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 19:55:07.398807 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:07.398770 2574 patch_prober.go:28] interesting pod/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 23 19:55:07.399239 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:07.398830 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 23 19:55:08.402230 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.402197 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:55:08.402686 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.402494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:55:08.403420 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.403396 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 19:55:08.980332 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.980302 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc"] Jul 23 19:55:08.984245 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.984219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:08.987407 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.987386 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-459a5a\"/\"openshift-service-ca.crt\"" Jul 23 19:55:08.987550 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.987455 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-459a5a\"/\"kube-root-ca.crt\"" Jul 23 19:55:08.987550 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.987531 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-459a5a\"/\"default-dockercfg-bjr7g\"" Jul 23 19:55:08.987647 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.987596 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-459a5a\"/\"auth-test-model-459a5a-kserve-self-signed-certs\"" Jul 23 19:55:08.996094 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:08.996071 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc"] Jul 23 19:55:09.014282 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.014247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tls-certs\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.014406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.014298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tmp-dir\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.014406 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.014382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-dshm\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.014482 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.014419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhkf\" (UniqueName: \"kubernetes.io/projected/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-kube-api-access-6vhkf\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.014482 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.014456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-model-cache\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.014568 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.014544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-home\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.115507 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.115462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-dshm\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.115507 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.115514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhkf\" (UniqueName: \"kubernetes.io/projected/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-kube-api-access-6vhkf\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.115834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.115550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-model-cache\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.115834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.115579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-home\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.115834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.115671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tls-certs\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.115834 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.115696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tmp-dir\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.116070 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.116040 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-model-cache\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.116070 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.116065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tmp-dir\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.116243 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.116107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-home\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.117922 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.117894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-dshm\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.118196 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.118179 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tls-certs\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.124470 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.124449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhkf\" (UniqueName: \"kubernetes.io/projected/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-kube-api-access-6vhkf\") pod \"auth-test-model-459a5a-kserve-778c97b699-pn2tc\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.294666 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.294576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:09.434754 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:09.434704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc"] Jul 23 19:55:09.437125 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:55:09.437097 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68e8c8d8_88b4_4676_8a01_5d5342e25e0c.slice/crio-7ad4513f31d5e89afabe5af6e95da3a9133d648323022f41cbe3c16466631a45 WatchSource:0}: Error finding container 7ad4513f31d5e89afabe5af6e95da3a9133d648323022f41cbe3c16466631a45: Status 404 returned error can't find the container with id 7ad4513f31d5e89afabe5af6e95da3a9133d648323022f41cbe3c16466631a45 Jul 23 19:55:10.192180 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:10.192141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" event={"ID":"68e8c8d8-88b4-4676-8a01-5d5342e25e0c","Type":"ContainerStarted","Data":"33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf"} Jul 23 19:55:10.192180 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:10.192191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" event={"ID":"68e8c8d8-88b4-4676-8a01-5d5342e25e0c","Type":"ContainerStarted","Data":"7ad4513f31d5e89afabe5af6e95da3a9133d648323022f41cbe3c16466631a45"} Jul 23 19:55:10.209661 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:10.209610 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" podStartSLOduration=2.209592806 podStartE2EDuration="2.209592806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:55:10.208332448 +0000 UTC m=+2357.780952978" watchObservedRunningTime="2026-07-23 19:55:10.209592806 +0000 UTC m=+2357.782213347" Jul 23 19:55:19.295789 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:19.295748 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:19.296296 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:19.295820 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:19.308525 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:19.308496 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:20.256135 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:20.256103 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:55:52.982692 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:52.982665 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:55:52.998240 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:55:52.998211 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 19:57:23.054334 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:23.054289 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc"] Jul 23 19:57:38.074375 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.074320 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" podUID="68e8c8d8-88b4-4676-8a01-5d5342e25e0c" containerName="main" containerID="cri-o://33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf" gracePeriod=45 Jul 23 19:57:38.327713 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.327647 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.368268 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-model-cache\") pod \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.368336 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-home\") pod \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.368374 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vhkf\" (UniqueName: \"kubernetes.io/projected/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-kube-api-access-6vhkf\") pod \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.368444 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tmp-dir\") pod \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.368487 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tls-certs\") pod \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.368528 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-dshm\") pod \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\" (UID: \"68e8c8d8-88b4-4676-8a01-5d5342e25e0c\") " Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.369774 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "68e8c8d8-88b4-4676-8a01-5d5342e25e0c" (UID: "68e8c8d8-88b4-4676-8a01-5d5342e25e0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.369972 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-home" (OuterVolumeSpecName: "home") pod "68e8c8d8-88b4-4676-8a01-5d5342e25e0c" (UID: "68e8c8d8-88b4-4676-8a01-5d5342e25e0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:57:38.370777 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.370476 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68e8c8d8-88b4-4676-8a01-5d5342e25e0c" (UID: "68e8c8d8-88b4-4676-8a01-5d5342e25e0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:57:38.377841 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.377770 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-kube-api-access-6vhkf" (OuterVolumeSpecName: "kube-api-access-6vhkf") pod "68e8c8d8-88b4-4676-8a01-5d5342e25e0c" (UID: "68e8c8d8-88b4-4676-8a01-5d5342e25e0c"). InnerVolumeSpecName "kube-api-access-6vhkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:57:38.377841 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.377775 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-dshm" (OuterVolumeSpecName: "dshm") pod "68e8c8d8-88b4-4676-8a01-5d5342e25e0c" (UID: "68e8c8d8-88b4-4676-8a01-5d5342e25e0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:57:38.378093 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.378040 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68e8c8d8-88b4-4676-8a01-5d5342e25e0c" (UID: "68e8c8d8-88b4-4676-8a01-5d5342e25e0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:57:38.469514 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.469479 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-model-cache\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:57:38.469514 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.469510 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-home\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:57:38.469752 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.469523 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhkf\" (UniqueName: \"kubernetes.io/projected/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-kube-api-access-6vhkf\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:57:38.469752 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.469534 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tmp-dir\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:57:38.469752 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.469544 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-tls-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:57:38.469752 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.469553 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e8c8d8-88b4-4676-8a01-5d5342e25e0c-dshm\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:57:38.994650 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.994613 2574 generic.go:356] "Generic (PLEG): container finished" podID="68e8c8d8-88b4-4676-8a01-5d5342e25e0c" containerID="33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf" exitCode=0 Jul 23 19:57:38.994854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.994681 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" Jul 23 19:57:38.994854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.994697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" event={"ID":"68e8c8d8-88b4-4676-8a01-5d5342e25e0c","Type":"ContainerDied","Data":"33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf"} Jul 23 19:57:38.994854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.994757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc" event={"ID":"68e8c8d8-88b4-4676-8a01-5d5342e25e0c","Type":"ContainerDied","Data":"7ad4513f31d5e89afabe5af6e95da3a9133d648323022f41cbe3c16466631a45"} Jul 23 19:57:38.994854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:38.994776 2574 scope.go:117] "RemoveContainer" containerID="33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf" Jul 23 19:57:39.003579 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:39.003562 2574 scope.go:117] "RemoveContainer" containerID="33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf" Jul 23 19:57:39.003864 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:57:39.003846 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf\": container with ID starting with 33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf not found: ID does not exist" containerID="33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf" Jul 23 19:57:39.003921 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:39.003872 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf"} err="failed to get container status \"33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf\": rpc error: code = NotFound desc = could not find container \"33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf\": container with ID starting with 33839825413ba1bd8c3b2a08e59e63873c4829b0057c34698a52262a104134bf not found: ID does not exist" Jul 23 19:57:39.012081 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:39.012058 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc"] Jul 23 19:57:39.015376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:39.015355 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-459a5a/auth-test-model-459a5a-kserve-778c97b699-pn2tc"] Jul 23 19:57:40.848184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:57:40.848152 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e8c8d8-88b4-4676-8a01-5d5342e25e0c" path="/var/lib/kubelet/pods/68e8c8d8-88b4-4676-8a01-5d5342e25e0c/volumes" Jul 23 19:58:07.985177 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.985145 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v"] Jul 23 19:58:07.985614 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.985547 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e8c8d8-88b4-4676-8a01-5d5342e25e0c" containerName="main" Jul 23 19:58:07.985614 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.985559 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e8c8d8-88b4-4676-8a01-5d5342e25e0c" containerName="main" Jul 23 19:58:07.985691 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.985631 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e8c8d8-88b4-4676-8a01-5d5342e25e0c" containerName="main" Jul 23 19:58:07.988725 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.988708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:07.992403 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.992378 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"default-dockercfg-zzhln\"" Jul 23 19:58:07.992403 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.992392 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"kube-root-ca.crt\"" Jul 23 19:58:07.992598 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.992381 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"openshift-service-ca.crt\"" Jul 23 19:58:07.992598 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.992381 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"sub-test-model-829079-kserve-self-signed-certs\"" Jul 23 19:58:07.997368 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:07.997349 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v"] Jul 23 19:58:08.142396 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.142354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-dshm\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.142587 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.142460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tmp-dir\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.142587 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.142505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tls-certs\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.142587 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.142543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zt8\" (UniqueName: \"kubernetes.io/projected/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-kube-api-access-56zt8\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.142697 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.142617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-model-cache\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.142697 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.142659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-home\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.243935 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.243841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zt8\" (UniqueName: \"kubernetes.io/projected/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-kube-api-access-56zt8\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.243935 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.243892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-model-cache\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.243935 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.243937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-home\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.244255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.243968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-dshm\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.244255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.244045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tmp-dir\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.244255 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.244077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tls-certs\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.244437 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.244410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-model-cache\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.244501 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.244466 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-home\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.244565 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.244539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tmp-dir\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.246581 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.246556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-dshm\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.246841 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.246821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tls-certs\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.252133 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.252110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zt8\" (UniqueName: \"kubernetes.io/projected/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-kube-api-access-56zt8\") pod \"sub-test-model-829079-kserve-74b78f5fc7-kc59v\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.298956 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.298921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:08.452282 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:08.452245 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v"] Jul 23 19:58:08.454411 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:58:08.454383 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19cf96c1_ae50_42ad_9232_c47b5ade6e7e.slice/crio-2c13952c554b9af1ad37db261f266c5d9eabdcc563205d354be018400505d7e8 WatchSource:0}: Error finding container 2c13952c554b9af1ad37db261f266c5d9eabdcc563205d354be018400505d7e8: Status 404 returned error can't find the container with id 2c13952c554b9af1ad37db261f266c5d9eabdcc563205d354be018400505d7e8 Jul 23 19:58:09.166348 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:09.166319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" event={"ID":"19cf96c1-ae50-42ad-9232-c47b5ade6e7e","Type":"ContainerStarted","Data":"865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3"} Jul 23 19:58:09.166776 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:09.166358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" event={"ID":"19cf96c1-ae50-42ad-9232-c47b5ade6e7e","Type":"ContainerStarted","Data":"2c13952c554b9af1ad37db261f266c5d9eabdcc563205d354be018400505d7e8"} Jul 23 19:58:09.186652 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:09.186606 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" podStartSLOduration=2.186590306 podStartE2EDuration="2.186590306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:58:09.183938536 +0000 UTC m=+2536.756559086" watchObservedRunningTime="2026-07-23 19:58:09.186590306 +0000 UTC m=+2536.759210911" Jul 23 19:58:18.300035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:18.299986 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:18.300489 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:18.300241 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:18.312871 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:18.312836 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:19.239124 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:19.239093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:43.383572 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:43.383530 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v"] Jul 23 19:58:58.404798 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.404720 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" podUID="19cf96c1-ae50-42ad-9232-c47b5ade6e7e" containerName="main" containerID="cri-o://865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3" gracePeriod=45 Jul 23 19:58:58.681394 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.681368 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:58.834128 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834087 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-model-cache\") pod \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " Jul 23 19:58:58.834328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834166 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56zt8\" (UniqueName: \"kubernetes.io/projected/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-kube-api-access-56zt8\") pod \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " Jul 23 19:58:58.834328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-home\") pod \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " Jul 23 19:58:58.834328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834270 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tls-certs\") pod \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " Jul 23 19:58:58.834328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834296 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tmp-dir\") pod \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " Jul 23 19:58:58.834328 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834319 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-dshm\") pod \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\" (UID: \"19cf96c1-ae50-42ad-9232-c47b5ade6e7e\") " Jul 23 19:58:58.834601 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834433 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "19cf96c1-ae50-42ad-9232-c47b5ade6e7e" (UID: "19cf96c1-ae50-42ad-9232-c47b5ade6e7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:58:58.834660 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834610 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-model-cache\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:58:58.834854 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834806 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-home" (OuterVolumeSpecName: "home") pod "19cf96c1-ae50-42ad-9232-c47b5ade6e7e" (UID: "19cf96c1-ae50-42ad-9232-c47b5ade6e7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:58:58.834973 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.834870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19cf96c1-ae50-42ad-9232-c47b5ade6e7e" (UID: "19cf96c1-ae50-42ad-9232-c47b5ade6e7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:58:58.836631 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.836608 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-dshm" (OuterVolumeSpecName: "dshm") pod "19cf96c1-ae50-42ad-9232-c47b5ade6e7e" (UID: "19cf96c1-ae50-42ad-9232-c47b5ade6e7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:58:58.836900 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.836884 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19cf96c1-ae50-42ad-9232-c47b5ade6e7e" (UID: "19cf96c1-ae50-42ad-9232-c47b5ade6e7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:58:58.836978 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.836958 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-kube-api-access-56zt8" (OuterVolumeSpecName: "kube-api-access-56zt8") pod "19cf96c1-ae50-42ad-9232-c47b5ade6e7e" (UID: "19cf96c1-ae50-42ad-9232-c47b5ade6e7e"). InnerVolumeSpecName "kube-api-access-56zt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:58:58.936171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.936064 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-home\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:58:58.936171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.936095 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tls-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:58:58.936171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.936105 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-tmp-dir\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:58:58.936171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.936113 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-dshm\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:58:58.936171 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:58.936121 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zt8\" (UniqueName: \"kubernetes.io/projected/19cf96c1-ae50-42ad-9232-c47b5ade6e7e-kube-api-access-56zt8\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 19:58:59.438369 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.438335 2574 generic.go:356] "Generic (PLEG): container finished" podID="19cf96c1-ae50-42ad-9232-c47b5ade6e7e" containerID="865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3" exitCode=0 Jul 23 19:58:59.438800 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.438379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" event={"ID":"19cf96c1-ae50-42ad-9232-c47b5ade6e7e","Type":"ContainerDied","Data":"865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3"} Jul 23 19:58:59.438800 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.438386 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" Jul 23 19:58:59.438800 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.438412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v" event={"ID":"19cf96c1-ae50-42ad-9232-c47b5ade6e7e","Type":"ContainerDied","Data":"2c13952c554b9af1ad37db261f266c5d9eabdcc563205d354be018400505d7e8"} Jul 23 19:58:59.438800 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.438430 2574 scope.go:117] "RemoveContainer" containerID="865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3" Jul 23 19:58:59.447561 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.447539 2574 scope.go:117] "RemoveContainer" containerID="865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3" Jul 23 19:58:59.447874 ip-10-0-133-134 kubenswrapper[2574]: E0723 19:58:59.447853 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3\": container with ID starting with 865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3 not found: ID does not exist" containerID="865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3" Jul 23 19:58:59.447959 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.447888 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3"} err="failed to get container status \"865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3\": rpc error: code = NotFound desc = could not find container \"865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3\": container with ID starting with 865b5236dd1609ee9a67cc89ec62f56d77cf4470bbee0dcc01e5ae5f7adaffa3 not found: ID does not exist" Jul 23 19:58:59.455385 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.455356 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v"] Jul 23 19:58:59.458634 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:58:59.458611 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-829079/sub-test-model-829079-kserve-74b78f5fc7-kc59v"] Jul 23 19:59:00.848134 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:00.848093 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cf96c1-ae50-42ad-9232-c47b5ade6e7e" path="/var/lib/kubelet/pods/19cf96c1-ae50-42ad-9232-c47b5ade6e7e/volumes" Jul 23 19:59:41.281242 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.281206 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz"] Jul 23 19:59:41.281710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.281614 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cf96c1-ae50-42ad-9232-c47b5ade6e7e" containerName="main" Jul 23 19:59:41.281710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.281625 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cf96c1-ae50-42ad-9232-c47b5ade6e7e" containerName="main" Jul 23 19:59:41.281710 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.281694 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cf96c1-ae50-42ad-9232-c47b5ade6e7e" containerName="main" Jul 23 19:59:41.285154 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.285134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.288946 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.288915 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"kube-root-ca.crt\"" Jul 23 19:59:41.289058 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.289021 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"openshift-service-ca.crt\"" Jul 23 19:59:41.289125 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.289110 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"default-dockercfg-zzhln\"" Jul 23 19:59:41.289172 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.289123 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-829079\"/\"rate-test-model-829079-kserve-self-signed-certs\"" Jul 23 19:59:41.293035 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.293013 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz"] Jul 23 19:59:41.449193 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.449147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-model-cache\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.449376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.449202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t8nt\" (UniqueName: \"kubernetes.io/projected/402e6da8-660d-4966-8390-317a851f8de3-kube-api-access-8t8nt\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.449376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.449283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402e6da8-660d-4966-8390-317a851f8de3-tls-certs\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.449376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.449307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-home\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.449376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.449355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-dshm\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.449376 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.449376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-tmp-dir\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.549941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.549847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-home\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.549941 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.549934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-dshm\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.550184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.549953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-tmp-dir\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.550184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.550024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-model-cache\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.550184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.550050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t8nt\" (UniqueName: \"kubernetes.io/projected/402e6da8-660d-4966-8390-317a851f8de3-kube-api-access-8t8nt\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.550184 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.550111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402e6da8-660d-4966-8390-317a851f8de3-tls-certs\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.550402 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.550325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-home\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.550827 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.550799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-tmp-dir\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.551246 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.550841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-model-cache\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.552324 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.552300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-dshm\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.552747 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.552714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402e6da8-660d-4966-8390-317a851f8de3-tls-certs\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.559552 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.559531 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t8nt\" (UniqueName: \"kubernetes.io/projected/402e6da8-660d-4966-8390-317a851f8de3-kube-api-access-8t8nt\") pod \"rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.596191 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.596159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:41.742889 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:41.742860 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz"] Jul 23 19:59:41.744675 ip-10-0-133-134 kubenswrapper[2574]: W0723 19:59:41.744641 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod402e6da8_660d_4966_8390_317a851f8de3.slice/crio-5f56abf900e1510d05dab604a709024c0a8f494b1ab035391d9638fcbb7274e9 WatchSource:0}: Error finding container 5f56abf900e1510d05dab604a709024c0a8f494b1ab035391d9638fcbb7274e9: Status 404 returned error can't find the container with id 5f56abf900e1510d05dab604a709024c0a8f494b1ab035391d9638fcbb7274e9 Jul 23 19:59:42.666477 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:42.666442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" event={"ID":"402e6da8-660d-4966-8390-317a851f8de3","Type":"ContainerStarted","Data":"d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407"} Jul 23 19:59:42.666477 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:42.666484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" event={"ID":"402e6da8-660d-4966-8390-317a851f8de3","Type":"ContainerStarted","Data":"5f56abf900e1510d05dab604a709024c0a8f494b1ab035391d9638fcbb7274e9"} Jul 23 19:59:42.683747 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:42.683691 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" podStartSLOduration=1.683676699 podStartE2EDuration="1.683676699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 19:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:59:42.682011371 +0000 UTC m=+2630.254631922" watchObservedRunningTime="2026-07-23 19:59:42.683676699 +0000 UTC m=+2630.256297238" Jul 23 19:59:51.596822 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:51.596785 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:51.597340 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:51.597086 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:51.609979 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:51.609951 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 19:59:51.725479 ip-10-0-133-134 kubenswrapper[2574]: I0723 19:59:51.725450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 20:00:26.385199 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:26.385161 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz"] Jul 23 20:00:41.406520 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.406461 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" podUID="402e6da8-660d-4966-8390-317a851f8de3" containerName="main" containerID="cri-o://d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407" gracePeriod=45 Jul 23 20:00:41.674011 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.673981 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 20:00:41.722176 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722144 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t8nt\" (UniqueName: \"kubernetes.io/projected/402e6da8-660d-4966-8390-317a851f8de3-kube-api-access-8t8nt\") pod \"402e6da8-660d-4966-8390-317a851f8de3\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " Jul 23 20:00:41.722373 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-model-cache\") pod \"402e6da8-660d-4966-8390-317a851f8de3\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " Jul 23 20:00:41.722373 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722231 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-dshm\") pod \"402e6da8-660d-4966-8390-317a851f8de3\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " Jul 23 20:00:41.722373 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722281 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402e6da8-660d-4966-8390-317a851f8de3-tls-certs\") pod \"402e6da8-660d-4966-8390-317a851f8de3\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " Jul 23 20:00:41.722373 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722314 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-home\") pod \"402e6da8-660d-4966-8390-317a851f8de3\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " Jul 23 20:00:41.722373 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-tmp-dir\") pod \"402e6da8-660d-4966-8390-317a851f8de3\" (UID: \"402e6da8-660d-4966-8390-317a851f8de3\") " Jul 23 20:00:41.722627 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722581 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-model-cache" (OuterVolumeSpecName: "model-cache") pod "402e6da8-660d-4966-8390-317a851f8de3" (UID: "402e6da8-660d-4966-8390-317a851f8de3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:00:41.722683 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722626 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-home" (OuterVolumeSpecName: "home") pod "402e6da8-660d-4966-8390-317a851f8de3" (UID: "402e6da8-660d-4966-8390-317a851f8de3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:00:41.722825 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "402e6da8-660d-4966-8390-317a851f8de3" (UID: "402e6da8-660d-4966-8390-317a851f8de3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:00:41.722890 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722813 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-model-cache\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:00:41.722890 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.722858 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-home\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:00:41.724580 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.724556 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402e6da8-660d-4966-8390-317a851f8de3-kube-api-access-8t8nt" (OuterVolumeSpecName: "kube-api-access-8t8nt") pod "402e6da8-660d-4966-8390-317a851f8de3" (UID: "402e6da8-660d-4966-8390-317a851f8de3"). InnerVolumeSpecName "kube-api-access-8t8nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:00:41.724580 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.724562 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-dshm" (OuterVolumeSpecName: "dshm") pod "402e6da8-660d-4966-8390-317a851f8de3" (UID: "402e6da8-660d-4966-8390-317a851f8de3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:00:41.724748 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.724565 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402e6da8-660d-4966-8390-317a851f8de3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "402e6da8-660d-4966-8390-317a851f8de3" (UID: "402e6da8-660d-4966-8390-317a851f8de3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 20:00:41.823349 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.823310 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-dshm\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:00:41.823349 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.823341 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402e6da8-660d-4966-8390-317a851f8de3-tls-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:00:41.823349 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.823351 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402e6da8-660d-4966-8390-317a851f8de3-tmp-dir\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:00:41.823349 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:41.823360 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t8nt\" (UniqueName: \"kubernetes.io/projected/402e6da8-660d-4966-8390-317a851f8de3-kube-api-access-8t8nt\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:00:42.001857 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.001703 2574 generic.go:356] "Generic (PLEG): container finished" podID="402e6da8-660d-4966-8390-317a851f8de3" containerID="d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407" exitCode=0 Jul 23 20:00:42.001857 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.001764 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" Jul 23 20:00:42.001857 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.001769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" event={"ID":"402e6da8-660d-4966-8390-317a851f8de3","Type":"ContainerDied","Data":"d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407"} Jul 23 20:00:42.001857 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.001820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz" event={"ID":"402e6da8-660d-4966-8390-317a851f8de3","Type":"ContainerDied","Data":"5f56abf900e1510d05dab604a709024c0a8f494b1ab035391d9638fcbb7274e9"} Jul 23 20:00:42.001857 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.001839 2574 scope.go:117] "RemoveContainer" containerID="d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407" Jul 23 20:00:42.011215 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.011199 2574 scope.go:117] "RemoveContainer" containerID="d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407" Jul 23 20:00:42.011467 ip-10-0-133-134 kubenswrapper[2574]: E0723 20:00:42.011448 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407\": container with ID starting with d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407 not found: ID does not exist" containerID="d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407" Jul 23 20:00:42.011534 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.011476 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407"} err="failed to get container status \"d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407\": rpc error: code = NotFound desc = could not find container \"d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407\": container with ID starting with d4c25348762e3c4b84a3a34e9616cd94ba4c60be1820e8eb493a2ab7ac3db407 not found: ID does not exist" Jul 23 20:00:42.022165 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.022139 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz"] Jul 23 20:00:42.025797 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.025773 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-829079/rate-test-model-829079-kserve-5dfbdd6cf5-5wsmz"] Jul 23 20:00:42.847494 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:42.847463 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402e6da8-660d-4966-8390-317a851f8de3" path="/var/lib/kubelet/pods/402e6da8-660d-4966-8390-317a851f8de3/volumes" Jul 23 20:00:53.022941 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:53.022869 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 20:00:53.031749 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:00:53.031711 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 20:01:08.437207 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:08.437167 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d"] Jul 23 20:01:08.437815 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:08.437432 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" containerID="cri-o://8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0" gracePeriod=30 Jul 23 20:01:13.703887 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.703823 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 20:01:13.825425 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825393 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-token\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.825633 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825453 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-envoy\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.825633 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825486 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-certs\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.825633 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825508 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-socket\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.825633 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825541 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-data\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.825633 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825569 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istiod-ca-cert\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.825633 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825596 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfzm6\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-kube-api-access-mfzm6\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.826029 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825657 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-podinfo\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.826029 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.825693 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-credential-socket\") pod \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\" (UID: \"466f3f89-f60e-4c11-9900-f58d3f0a7d66\") " Jul 23 20:01:13.826138 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.826079 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:13.826193 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.826173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:13.826248 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.826188 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:13.826302 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.826276 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 20:01:13.826696 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.826647 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-data" (OuterVolumeSpecName: "istio-data") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:13.828093 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.828070 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:13.828205 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.828097 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-kube-api-access-mfzm6" (OuterVolumeSpecName: "kube-api-access-mfzm6") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "kube-api-access-mfzm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:01:13.828280 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.828254 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 20:01:13.828443 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.828419 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-token" (OuterVolumeSpecName: "istio-token") pod "466f3f89-f60e-4c11-9900-f58d3f0a7d66" (UID: "466f3f89-f60e-4c11-9900-f58d3f0a7d66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:01:13.927441 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927405 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927441 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927436 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927441 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927445 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927708 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927454 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927708 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927467 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927708 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927475 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfzm6\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-kube-api-access-mfzm6\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927708 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927483 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927708 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927491 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/466f3f89-f60e-4c11-9900-f58d3f0a7d66-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:13.927708 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:13.927501 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/466f3f89-f60e-4c11-9900-f58d3f0a7d66-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:14.180413 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.180380 2574 generic.go:356] "Generic (PLEG): container finished" podID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerID="8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0" exitCode=0 Jul 23 20:01:14.180589 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.180427 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" Jul 23 20:01:14.180589 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.180448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" event={"ID":"466f3f89-f60e-4c11-9900-f58d3f0a7d66","Type":"ContainerDied","Data":"8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0"} Jul 23 20:01:14.180589 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.180498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d" event={"ID":"466f3f89-f60e-4c11-9900-f58d3f0a7d66","Type":"ContainerDied","Data":"a9fe377e62ceec424f74e56d6fd7d1504be5da280b7dd0fee4347f5de3c9d162"} Jul 23 20:01:14.180589 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.180522 2574 scope.go:117] "RemoveContainer" containerID="8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0" Jul 23 20:01:14.190332 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.190311 2574 scope.go:117] "RemoveContainer" containerID="684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890" Jul 23 20:01:14.197983 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.197964 2574 scope.go:117] "RemoveContainer" containerID="8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0" Jul 23 20:01:14.198225 ip-10-0-133-134 kubenswrapper[2574]: E0723 20:01:14.198209 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0\": container with ID starting with 8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0 not found: ID does not exist" containerID="8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0" Jul 23 20:01:14.198274 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.198234 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0"} err="failed to get container status \"8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0\": rpc error: code = NotFound desc = could not find container \"8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0\": container with ID starting with 8b1da23e29632c3b81ae4f4c5ed39f132b168b42cf1f33097c85d72f5f4395e0 not found: ID does not exist" Jul 23 20:01:14.198274 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.198250 2574 scope.go:117] "RemoveContainer" containerID="684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890" Jul 23 20:01:14.198465 ip-10-0-133-134 kubenswrapper[2574]: E0723 20:01:14.198446 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890\": container with ID starting with 684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890 not found: ID does not exist" containerID="684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890" Jul 23 20:01:14.198509 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.198472 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890"} err="failed to get container status \"684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890\": rpc error: code = NotFound desc = could not find container \"684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890\": container with ID starting with 684d8660e771e083a016879777dfabdac83d6f8365254d0c706eba9e4af1f890 not found: ID does not exist" Jul 23 20:01:14.203151 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.203131 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d"] Jul 23 20:01:14.206716 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.206690 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-459a5a-openshift-default-5f95b7d647-p7c8d"] Jul 23 20:01:14.854298 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:14.854254 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" path="/var/lib/kubelet/pods/466f3f89-f60e-4c11-9900-f58d3f0a7d66/volumes" Jul 23 20:01:30.259126 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:30.259088 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5"] Jul 23 20:01:30.259614 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:30.259346 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" containerID="cri-o://70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91" gracePeriod=30 Jul 23 20:01:30.965306 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:30.965273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8b59765-h7phg_21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff/authorino/0.log" Jul 23 20:01:35.521090 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.521018 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 20:01:35.618431 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-certs\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618431 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-data\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618454 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-credential-socket\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618481 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3ba37c-da24-4343-ba93-548952445a34-istiod-ca-cert\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618530 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-envoy\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618555 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3ba37c-da24-4343-ba93-548952445a34-istio-podinfo\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618574 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-socket\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618601 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-istio-token\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.618692 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618630 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrcld\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-kube-api-access-rrcld\") pod \"cf3ba37c-da24-4343-ba93-548952445a34\" (UID: \"cf3ba37c-da24-4343-ba93-548952445a34\") " Jul 23 20:01:35.619101 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618714 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:35.619101 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618805 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:35.619101 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.618989 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-certs\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.619101 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.619012 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-credential-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.619101 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.619047 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:35.619101 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.619068 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3ba37c-da24-4343-ba93-548952445a34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 20:01:35.619466 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.619441 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:35.620888 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.620859 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf3ba37c-da24-4343-ba93-548952445a34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 20:01:35.621045 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.621023 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:01:35.621250 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.621226 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 20:01:35.621331 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.621254 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-kube-api-access-rrcld" (OuterVolumeSpecName: "kube-api-access-rrcld") pod "cf3ba37c-da24-4343-ba93-548952445a34" (UID: "cf3ba37c-da24-4343-ba93-548952445a34"). InnerVolumeSpecName "kube-api-access-rrcld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 20:01:35.720205 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720170 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-envoy\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.720205 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720198 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3ba37c-da24-4343-ba93-548952445a34-istio-podinfo\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.720205 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720207 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-workload-socket\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.720205 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720216 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-istio-token\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.720494 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720225 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcld\" (UniqueName: \"kubernetes.io/projected/cf3ba37c-da24-4343-ba93-548952445a34-kube-api-access-rrcld\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.720494 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720234 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3ba37c-da24-4343-ba93-548952445a34-istio-data\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:35.720494 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:35.720242 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3ba37c-da24-4343-ba93-548952445a34-istiod-ca-cert\") on node \"ip-10-0-133-134.ec2.internal\" DevicePath \"\"" Jul 23 20:01:36.173788 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.173759 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9622c_7f3b62be-cb47-4457-bf01-9159b658eb36/manager/0.log" Jul 23 20:01:36.296925 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.296884 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf3ba37c-da24-4343-ba93-548952445a34" containerID="70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91" exitCode=0 Jul 23 20:01:36.297103 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.296944 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" Jul 23 20:01:36.297103 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.296969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" event={"ID":"cf3ba37c-da24-4343-ba93-548952445a34","Type":"ContainerDied","Data":"70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91"} Jul 23 20:01:36.297103 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.297026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5" event={"ID":"cf3ba37c-da24-4343-ba93-548952445a34","Type":"ContainerDied","Data":"9ffefe05e0949cfd267660b587bd971365a1fba123942a4ae891f7df791e5e90"} Jul 23 20:01:36.297103 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.297050 2574 scope.go:117] "RemoveContainer" containerID="70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91" Jul 23 20:01:36.307118 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.307098 2574 scope.go:117] "RemoveContainer" containerID="c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a" Jul 23 20:01:36.315140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.315117 2574 scope.go:117] "RemoveContainer" containerID="70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91" Jul 23 20:01:36.315424 ip-10-0-133-134 kubenswrapper[2574]: E0723 20:01:36.315404 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91\": container with ID starting with 70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91 not found: ID does not exist" containerID="70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91" Jul 23 20:01:36.315487 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.315435 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91"} err="failed to get container status \"70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91\": rpc error: code = NotFound desc = could not find container \"70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91\": container with ID starting with 70a26f2c5c66ce79c8ba6dbcb015e64875c19e6184f60081a1be53f5d2357c91 not found: ID does not exist" Jul 23 20:01:36.315487 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.315454 2574 scope.go:117] "RemoveContainer" containerID="c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a" Jul 23 20:01:36.315699 ip-10-0-133-134 kubenswrapper[2574]: E0723 20:01:36.315683 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a\": container with ID starting with c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a not found: ID does not exist" containerID="c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a" Jul 23 20:01:36.315761 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.315706 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a"} err="failed to get container status \"c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a\": rpc error: code = NotFound desc = could not find container \"c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a\": container with ID starting with c5b54a9a3e069befaebdd129b8de257a04c80323280f0295922370706eb8ec6a not found: ID does not exist" Jul 23 20:01:36.319484 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.319463 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5"] Jul 23 20:01:36.323229 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.323206 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-829079-openshift-default-66c6c8c799-mt9z5"] Jul 23 20:01:36.411928 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.411883 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8fd995c4-wlzdx_7c551bba-efdf-4b2d-a806-f98d7eafa6f9/manager/0.log" Jul 23 20:01:36.518018 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.517933 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pzbbn_795b8eaf-8515-4fc4-82cf-dfbfc56b09f1/server/0.log" Jul 23 20:01:36.639510 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.639477 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vdvc7_60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee/manager/0.log" Jul 23 20:01:36.847665 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.847579 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3ba37c-da24-4343-ba93-548952445a34" path="/var/lib/kubelet/pods/cf3ba37c-da24-4343-ba93-548952445a34/volumes" Jul 23 20:01:36.871506 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.871476 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf4f8c896-cz7h7_4ddb66f1-c475-4d2d-a746-3a70f38e6856/manager/0.log" Jul 23 20:01:36.877855 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:36.877834 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf4f8c896-cz7h7_4ddb66f1-c475-4d2d-a746-3a70f38e6856/copy-manifests/0.log" Jul 23 20:01:37.334004 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:37.333967 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79b7dbcf5c-69gl7_621e05d1-6868-4488-a07e-87aa1f36b095/maas-api/0.log" Jul 23 20:01:37.444690 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:37.444658 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747250-h8dxv_a93abb7f-11f6-4e4b-9de6-766d5a38ad02/cleanup/0.log" Jul 23 20:01:37.767554 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:37.767511 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-k8tsq_e5209404-da52-42e3-845f-e1767d3ca080/postgres/0.log" Jul 23 20:01:39.192821 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:39.192782 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8b59765-h7phg_21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff/authorino/0.log" Jul 23 20:01:39.315387 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:39.315355 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-cxt9d_2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307/manager/0.log" Jul 23 20:01:39.531847 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:39.531763 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-tgd69_a0d629ce-0a9c-48c6-accd-87ab1e5e7685/kuadrant-console-plugin/0.log" Jul 23 20:01:39.655200 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:39.655164 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-59nfn_bc384ab1-501c-49b4-9dd6-8fa2b57f7e06/registry-server/0.log" Jul 23 20:01:39.779497 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:39.779450 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2sjwb_56a96f9c-a12c-44c4-9395-8215c34a6671/manager/0.log" Jul 23 20:01:40.014777 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:40.014746 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8zwxj_b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e/manager/0.log" Jul 23 20:01:40.359528 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:40.359439 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qjxcd_93987080-556b-44c8-9956-7e10bb23c22b/istio-proxy/0.log" Jul 23 20:01:40.690495 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:40.690472 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6474cf8df7-pcq4k_072438c0-50ff-4360-804e-fa57f5550652/kube-auth-proxy/0.log" Jul 23 20:01:40.807845 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:40.807806 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/1.log" Jul 23 20:01:40.928221 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:40.928187 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-npj46_6d3f684b-7997-44d1-a1d6-9946459a79be/payload-pre-processing/0.log" Jul 23 20:01:41.144872 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:41.144798 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64cbdc95dd-q7nph_10489313-35b1-47b5-a9b0-1e2df4a66bf6/router/0.log" Jul 23 20:01:47.715309 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715240 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ttxvk/must-gather-h2hwv"] Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715813 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715833 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715841 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715846 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715852 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715858 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715875 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402e6da8-660d-4966-8390-317a851f8de3" containerName="main" Jul 23 20:01:47.715886 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715880 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="402e6da8-660d-4966-8390-317a851f8de3" containerName="main" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715945 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715953 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715959 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="402e6da8-660d-4966-8390-317a851f8de3" containerName="main" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715967 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.715973 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3ba37c-da24-4343-ba93-548952445a34" containerName="istio-proxy" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.716043 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" Jul 23 20:01:47.716140 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.716048 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="466f3f89-f60e-4c11-9900-f58d3f0a7d66" containerName="istio-proxy" Jul 23 20:01:47.720690 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.720669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:47.724772 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.724749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ttxvk\"/\"openshift-service-ca.crt\"" Jul 23 20:01:47.724772 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.724762 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ttxvk\"/\"default-dockercfg-nlhhw\"" Jul 23 20:01:47.725568 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.725544 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ttxvk\"/\"kube-root-ca.crt\"" Jul 23 20:01:47.733812 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.733770 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttxvk/must-gather-h2hwv"] Jul 23 20:01:47.833255 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.833209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/809d9019-fb68-4ae4-8bed-1539c4cf3058-must-gather-output\") pod \"must-gather-h2hwv\" (UID: \"809d9019-fb68-4ae4-8bed-1539c4cf3058\") " pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:47.833454 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.833329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n97r\" (UniqueName: \"kubernetes.io/projected/809d9019-fb68-4ae4-8bed-1539c4cf3058-kube-api-access-2n97r\") pod \"must-gather-h2hwv\" (UID: \"809d9019-fb68-4ae4-8bed-1539c4cf3058\") " pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:47.934030 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.933986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n97r\" (UniqueName: \"kubernetes.io/projected/809d9019-fb68-4ae4-8bed-1539c4cf3058-kube-api-access-2n97r\") pod \"must-gather-h2hwv\" (UID: \"809d9019-fb68-4ae4-8bed-1539c4cf3058\") " pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:47.934218 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.934045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/809d9019-fb68-4ae4-8bed-1539c4cf3058-must-gather-output\") pod \"must-gather-h2hwv\" (UID: \"809d9019-fb68-4ae4-8bed-1539c4cf3058\") " pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:47.934378 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.934361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/809d9019-fb68-4ae4-8bed-1539c4cf3058-must-gather-output\") pod \"must-gather-h2hwv\" (UID: \"809d9019-fb68-4ae4-8bed-1539c4cf3058\") " pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:47.942429 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:47.942392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n97r\" (UniqueName: \"kubernetes.io/projected/809d9019-fb68-4ae4-8bed-1539c4cf3058-kube-api-access-2n97r\") pod \"must-gather-h2hwv\" (UID: \"809d9019-fb68-4ae4-8bed-1539c4cf3058\") " pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:48.031112 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:48.031017 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttxvk/must-gather-h2hwv" Jul 23 20:01:48.175195 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:48.175165 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttxvk/must-gather-h2hwv"] Jul 23 20:01:48.177081 ip-10-0-133-134 kubenswrapper[2574]: W0723 20:01:48.177046 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod809d9019_fb68_4ae4_8bed_1539c4cf3058.slice/crio-02d2e8224a45cf8442970a8b7e77d81b3061b15cf59327f3c88a6d1cf7b2b8d7 WatchSource:0}: Error finding container 02d2e8224a45cf8442970a8b7e77d81b3061b15cf59327f3c88a6d1cf7b2b8d7: Status 404 returned error can't find the container with id 02d2e8224a45cf8442970a8b7e77d81b3061b15cf59327f3c88a6d1cf7b2b8d7 Jul 23 20:01:48.178965 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:48.178944 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 20:01:48.369607 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:48.369520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttxvk/must-gather-h2hwv" event={"ID":"809d9019-fb68-4ae4-8bed-1539c4cf3058","Type":"ContainerStarted","Data":"02d2e8224a45cf8442970a8b7e77d81b3061b15cf59327f3c88a6d1cf7b2b8d7"} Jul 23 20:01:49.377566 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:49.377530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttxvk/must-gather-h2hwv" event={"ID":"809d9019-fb68-4ae4-8bed-1539c4cf3058","Type":"ContainerStarted","Data":"149d27bce45252d1068a7da3c7c81164030b342cf3285712afbc88eb691d0f2c"} Jul 23 20:01:49.378084 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:49.377580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttxvk/must-gather-h2hwv" event={"ID":"809d9019-fb68-4ae4-8bed-1539c4cf3058","Type":"ContainerStarted","Data":"f5e597d21fda345bf8ee2a0de853d96328f81607bb40ff36bd5ddbc3f3705758"} Jul 23 20:01:49.397104 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:49.397038 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ttxvk/must-gather-h2hwv" podStartSLOduration=1.404332856 podStartE2EDuration="2.39701508s" totalImagesPullingTime="992.682224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 20:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:01:49.393553979 +0000 UTC m=+2756.966174519" watchObservedRunningTime="2026-07-23 20:01:49.39701508 +0000 UTC m=+2756.969635621" Jul 23 20:01:50.782486 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:50.782439 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2gqgt_6b0171d6-8abc-4a3e-a72a-f29271153a70/global-pull-secret-syncer/0.log" Jul 23 20:01:50.952742 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:50.952676 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mfjtk_1abb1ad3-cc68-46d5-a07d-d2b4a3918de9/konnectivity-agent/0.log" Jul 23 20:01:51.009474 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:51.009432 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-134.ec2.internal_ade0de55707084dd778646963520c9b6/haproxy/0.log" Jul 23 20:01:56.837754 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:56.837558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8b59765-h7phg_21f2c2a2-d436-4a9a-8d4c-c5ba5c29baff/authorino/0.log" Jul 23 20:01:56.872190 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:56.872153 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-cxt9d_2abf4bb0-d5fe-4fd6-8ab0-d7f15f1dd307/manager/0.log" Jul 23 20:01:56.949544 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:56.949512 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-tgd69_a0d629ce-0a9c-48c6-accd-87ab1e5e7685/kuadrant-console-plugin/0.log" Jul 23 20:01:57.015875 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:57.015835 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-59nfn_bc384ab1-501c-49b4-9dd6-8fa2b57f7e06/registry-server/0.log" Jul 23 20:01:57.242012 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:57.241893 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2sjwb_56a96f9c-a12c-44c4-9395-8215c34a6671/manager/0.log" Jul 23 20:01:57.383410 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:57.383372 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8zwxj_b5b9f32d-53c1-4a1c-96df-f51b1d3ed34e/manager/0.log" Jul 23 20:01:59.284781 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:59.284726 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-kc7v2_6955e1ef-192b-4f01-9d94-ec04d7e747b6/cluster-monitoring-operator/0.log" Jul 23 20:01:59.604802 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:59.604694 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4l7b_269dcf6f-ff58-4a89-b4c5-db30038b1dcf/node-exporter/0.log" Jul 23 20:01:59.623648 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:59.623616 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4l7b_269dcf6f-ff58-4a89-b4c5-db30038b1dcf/kube-rbac-proxy/0.log" Jul 23 20:01:59.644594 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:01:59.644568 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4l7b_269dcf6f-ff58-4a89-b4c5-db30038b1dcf/init-textfile/0.log" Jul 23 20:02:00.893970 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.893934 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p"] Jul 23 20:02:00.903143 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.903104 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:00.906453 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.906414 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p"] Jul 23 20:02:00.976755 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.974363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-sys\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:00.976755 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.974421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-lib-modules\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:00.976755 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.974446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-podres\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:00.976755 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.974526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rjw\" (UniqueName: \"kubernetes.io/projected/a90816f9-0be6-454b-b6b4-4bee9df606d0-kube-api-access-49rjw\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:00.976755 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:00.974565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-proc\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075077 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-sys\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075265 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-lib-modules\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075265 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-sys\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075265 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-podres\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075265 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-lib-modules\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075430 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-podres\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075430 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rjw\" (UniqueName: \"kubernetes.io/projected/a90816f9-0be6-454b-b6b4-4bee9df606d0-kube-api-access-49rjw\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075430 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-proc\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.075601 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.075510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a90816f9-0be6-454b-b6b4-4bee9df606d0-proc\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.084393 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.084363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rjw\" (UniqueName: \"kubernetes.io/projected/a90816f9-0be6-454b-b6b4-4bee9df606d0-kube-api-access-49rjw\") pod \"perf-node-gather-daemonset-hf97p\" (UID: \"a90816f9-0be6-454b-b6b4-4bee9df606d0\") " pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.228621 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.228590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:01.431457 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.431409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p"] Jul 23 20:02:01.481185 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.481097 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" event={"ID":"a90816f9-0be6-454b-b6b4-4bee9df606d0","Type":"ContainerStarted","Data":"273bf5af11c0783cb13c1da09e14df9b8900c87a339e0715d5aec3b41da999be"} Jul 23 20:02:01.499131 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:01.499034 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-q9qgs_1db54a43-75dd-4d51-95ac-b95144a6f0fb/networking-console-plugin/0.log" Jul 23 20:02:02.212890 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:02.212766 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/1.log" Jul 23 20:02:02.221979 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:02.221944 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5zzxw_a909b563-360e-4e57-9755-36fcd9a261cc/console-operator/2.log" Jul 23 20:02:02.490625 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:02.490268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" event={"ID":"a90816f9-0be6-454b-b6b4-4bee9df606d0","Type":"ContainerStarted","Data":"b5e8ca78d775fe3696ae162c7046ff138104131dc1239b5e88c7b2a7a548d629"} Jul 23 20:02:02.491019 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:02.490638 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:02.509206 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:02.509146 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" podStartSLOduration=2.509124911 podStartE2EDuration="2.509124911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 20:02:02.505470911 +0000 UTC m=+2770.078091452" watchObservedRunningTime="2026-07-23 20:02:02.509124911 +0000 UTC m=+2770.081745455" Jul 23 20:02:03.299521 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:03.299489 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-hzgwq_6e34a4ee-f7cc-4171-ad03-75227ccbc291/volume-data-source-validator/0.log" Jul 23 20:02:04.445302 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:04.445256 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n4v72_21caab7f-0bf4-44bd-8a75-21db01239fb5/dns/0.log" Jul 23 20:02:04.466133 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:04.466109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n4v72_21caab7f-0bf4-44bd-8a75-21db01239fb5/kube-rbac-proxy/0.log" Jul 23 20:02:04.540233 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:04.540202 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qc484_36bf83e3-c725-41e9-b5b8-2b79ec57b7a7/dns-node-resolver/0.log" Jul 23 20:02:05.107379 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:05.107346 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xn7q_8d4319b6-feb5-4089-8896-3e93d2e2e099/node-ca/0.log" Jul 23 20:02:06.117500 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:06.117461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qjxcd_93987080-556b-44c8-9956-7e10bb23c22b/istio-proxy/0.log" Jul 23 20:02:06.287409 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:06.287378 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6474cf8df7-pcq4k_072438c0-50ff-4360-804e-fa57f5550652/kube-auth-proxy/0.log" Jul 23 20:02:06.318821 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:06.318774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/0.log" Jul 23 20:02:06.371017 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:06.370938 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/1.log" Jul 23 20:02:06.411441 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:06.411405 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-npj46_6d3f684b-7997-44d1-a1d6-9946459a79be/payload-pre-processing/0.log" Jul 23 20:02:06.465583 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:06.465548 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64cbdc95dd-q7nph_10489313-35b1-47b5-a9b0-1e2df4a66bf6/router/0.log" Jul 23 20:02:07.184970 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:07.184937 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hj9rw_24f25159-b9cc-4653-8ed2-d54b0a01f3e0/serve-healthcheck-canary/0.log" Jul 23 20:02:07.736395 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:07.736354 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-cr79v_e194a48c-e7dc-495e-a269-dc0a77b33f32/insights-operator/0.log" Jul 23 20:02:07.737291 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:07.737267 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-cr79v_e194a48c-e7dc-495e-a269-dc0a77b33f32/insights-operator/1.log" Jul 23 20:02:07.761246 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:07.761219 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6n8l4_f3869c9d-31a5-437f-96ae-a1f7c54f07cb/kube-rbac-proxy/0.log" Jul 23 20:02:07.785142 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:07.785113 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6n8l4_f3869c9d-31a5-437f-96ae-a1f7c54f07cb/exporter/0.log" Jul 23 20:02:07.810461 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:07.810433 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6n8l4_f3869c9d-31a5-437f-96ae-a1f7c54f07cb/extractor/0.log" Jul 23 20:02:08.508838 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:08.508810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ttxvk/perf-node-gather-daemonset-hf97p" Jul 23 20:02:10.371795 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:10.371699 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9622c_7f3b62be-cb47-4457-bf01-9159b658eb36/manager/0.log" Jul 23 20:02:10.765902 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:10.765864 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8fd995c4-wlzdx_7c551bba-efdf-4b2d-a806-f98d7eafa6f9/manager/0.log" Jul 23 20:02:10.789418 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:10.789387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pzbbn_795b8eaf-8515-4fc4-82cf-dfbfc56b09f1/server/0.log" Jul 23 20:02:10.910755 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:10.910698 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vdvc7_60f9aa80-8dfc-467a-bfdc-ec2d2e6018ee/manager/0.log" Jul 23 20:02:11.035433 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:11.035401 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf4f8c896-cz7h7_4ddb66f1-c475-4d2d-a746-3a70f38e6856/manager/0.log" Jul 23 20:02:11.062092 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:11.062058 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bf4f8c896-cz7h7_4ddb66f1-c475-4d2d-a746-3a70f38e6856/copy-manifests/0.log" Jul 23 20:02:12.070686 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.070653 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qjxcd_93987080-556b-44c8-9956-7e10bb23c22b/istio-proxy/0.log" Jul 23 20:02:12.234201 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.234172 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6474cf8df7-pcq4k_072438c0-50ff-4360-804e-fa57f5550652/kube-auth-proxy/0.log" Jul 23 20:02:12.257122 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.257092 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/0.log" Jul 23 20:02:12.316403 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.316369 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/1.log" Jul 23 20:02:12.349001 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.348895 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-npj46_6d3f684b-7997-44d1-a1d6-9946459a79be/payload-pre-processing/0.log" Jul 23 20:02:12.382907 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.382875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64cbdc95dd-q7nph_10489313-35b1-47b5-a9b0-1e2df4a66bf6/router/0.log" Jul 23 20:02:12.428513 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:12.428481 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kcp4k_3ae054ab-1776-4912-aa5f-b326a1b715b6/openshift-lws-operator/0.log" Jul 23 20:02:13.565398 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:13.565367 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qjxcd_93987080-556b-44c8-9956-7e10bb23c22b/istio-proxy/0.log" Jul 23 20:02:13.707252 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:13.707218 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6474cf8df7-pcq4k_072438c0-50ff-4360-804e-fa57f5550652/kube-auth-proxy/0.log" Jul 23 20:02:13.731525 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:13.730933 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/0.log" Jul 23 20:02:13.790811 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:13.790774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-wwjfn_f0271634-a745-49c3-a960-5e182d11e8de/istio-proxy/1.log" Jul 23 20:02:13.822012 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:13.821926 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-npj46_6d3f684b-7997-44d1-a1d6-9946459a79be/payload-pre-processing/0.log" Jul 23 20:02:13.865477 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:13.865444 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64cbdc95dd-q7nph_10489313-35b1-47b5-a9b0-1e2df4a66bf6/router/0.log" Jul 23 20:02:17.752588 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:17.752485 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-nmxnw_a8b21220-6461-4e91-9760-27f3914bf6d4/kube-storage-version-migrator-operator/1.log" Jul 23 20:02:17.754320 ip-10-0-133-134 kubenswrapper[2574]: I0723 20:02:17.754297 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-nmxnw_a8b21220-6461-4e91-9760-27f3914bf6d4/kube-storage-version-migrator-operator/0.log"