Aug 04 00:02:40.172743 ip-10-0-134-15 systemd[1]: Starting Kubernetes Kubelet... Aug 04 00:02:40.607245 ip-10-0-134-15 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:40.607245 ip-10-0-134-15 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 00:02:40.607245 ip-10-0-134-15 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:40.607245 ip-10-0-134-15 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 00:02:40.607245 ip-10-0-134-15 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 00:02:40.608179 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.608090 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 00:02:40.612417 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612400 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:40.612417 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612417 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612423 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612426 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612429 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612432 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612435 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612439 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612443 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612446 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612450 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612452 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612456 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612473 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612478 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612482 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612486 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612489 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612492 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612494 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:40.612508 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612497 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612500 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612502 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612505 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612508 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612515 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612518 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612521 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612524 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612526 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612529 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612531 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612534 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612537 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612539 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612542 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612544 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612547 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612549 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612551 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:40.612965 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612554 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612556 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612559 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612561 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612564 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612568 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612571 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612574 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612577 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612581 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612585 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612587 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612590 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612592 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612596 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612598 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612601 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612603 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612606 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:40.613445 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612609 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612612 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612614 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612617 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612619 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612622 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612625 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612628 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612630 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612634 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612638 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612641 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612643 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612646 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612648 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612651 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612653 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612656 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612665 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612668 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:40.613931 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612670 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612673 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612677 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612680 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612683 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612685 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612688 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612692 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612695 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612697 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612700 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612702 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612705 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612708 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612710 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612713 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612715 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612718 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612721 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.612723 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:40.614410 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614373 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614381 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614385 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614388 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614391 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614394 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614397 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614400 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614403 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614406 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614409 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614413 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614416 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614419 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614422 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614425 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614428 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614430 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614433 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:40.614917 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614436 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614439 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614443 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614445 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614448 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614451 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614453 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614456 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614473 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614476 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614478 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614481 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614483 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614486 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614488 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614492 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614495 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614513 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614516 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:40.615400 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614519 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614522 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614526 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614528 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614531 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614535 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614538 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614541 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614543 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614546 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614549 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614551 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614553 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614556 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614559 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614561 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614564 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614566 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614569 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:40.615921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614571 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614574 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614576 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614579 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614581 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614584 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614586 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614590 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614592 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614595 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614598 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614601 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614603 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614606 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614609 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614611 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614614 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614616 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614619 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614621 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:40.616389 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614624 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614627 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614629 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614632 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614636 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614640 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614643 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614646 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614656 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614659 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614662 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614664 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614667 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614671 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614674 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614677 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614680 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614682 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614685 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:40.616896 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614688 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614691 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.614694 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615076 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615081 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615083 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615086 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615089 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615092 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615095 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615097 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615100 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615103 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615106 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615108 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615111 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615114 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615116 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615119 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615121 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:40.617382 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615124 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615127 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615129 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615132 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615135 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615137 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615140 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615142 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615145 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615147 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615150 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615152 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615155 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615157 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615161 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615163 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615166 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615169 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615171 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615174 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:40.617895 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615176 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615179 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615182 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615184 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615187 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615190 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615194 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615197 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615200 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615203 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615205 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615208 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615211 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615213 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615216 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615218 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615220 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615224 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615226 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:40.618436 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615229 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615231 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615234 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615236 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615239 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615242 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615247 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615249 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615252 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615254 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615257 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615259 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615262 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615265 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615267 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615269 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615272 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615275 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615277 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615279 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:40.618935 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615282 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615285 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615287 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615289 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615292 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615294 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615297 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615299 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615302 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615306 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615309 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615312 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615314 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615317 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615319 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615322 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615325 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615327 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615331 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:40.619427 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615334 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615339 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615342 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.615345 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615418 2571 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615425 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615431 2571 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615436 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615440 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615443 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615447 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615475 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615480 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615483 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615487 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615490 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615493 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615496 2571 flags.go:64] FLAG: --cgroup-root="" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615499 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615502 2571 flags.go:64] FLAG: --client-ca-file="" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615505 2571 flags.go:64] FLAG: --cloud-provider="external" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615508 2571 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615511 2571 flags.go:64] FLAG: --cluster-domain="" Aug 04 00:02:40.619973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615515 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615518 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615521 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615524 2571 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615527 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615530 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615533 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615536 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615540 2571 flags.go:64] FLAG: --contention-profiling="false" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615543 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615546 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615549 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615552 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615556 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615559 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615562 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615564 2571 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615567 2571 flags.go:64] FLAG: --enable-server="true" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615570 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615575 2571 flags.go:64] FLAG: --event-burst="100" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615578 2571 flags.go:64] FLAG: --event-qps="50" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615580 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615583 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615586 2571 flags.go:64] FLAG: --eviction-hard="" Aug 04 00:02:40.620543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615590 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615593 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615596 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615599 2571 flags.go:64] FLAG: --eviction-soft="" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615601 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615604 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615607 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615610 2571 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615612 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615615 2571 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615618 2571 flags.go:64] FLAG: --feature-gates="" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615621 2571 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615624 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615627 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615631 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615633 2571 flags.go:64] FLAG: --healthz-port="10248" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615637 2571 flags.go:64] FLAG: --help="false" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615640 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-134-15.ec2.internal" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615643 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615646 2571 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615649 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615652 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615656 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615658 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 00:02:40.621115 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615661 2571 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615664 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615667 2571 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615670 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615673 2571 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615676 2571 flags.go:64] FLAG: --kube-reserved="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615678 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615681 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615684 2571 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615687 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615690 2571 flags.go:64] FLAG: --lock-file="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615693 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615696 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615699 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615704 2571 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615707 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615710 2571 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615713 2571 flags.go:64] FLAG: --logging-format="text" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615715 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615719 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615721 2571 flags.go:64] FLAG: --manifest-url="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615726 2571 flags.go:64] FLAG: --manifest-url-header="" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615730 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615733 2571 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615738 2571 flags.go:64] FLAG: --max-pods="110" Aug 04 00:02:40.621821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615741 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615744 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615747 2571 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615749 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615752 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615755 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615758 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615765 2571 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615768 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615771 2571 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615774 2571 flags.go:64] FLAG: --pod-cidr="" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615777 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615782 2571 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615785 2571 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615788 2571 flags.go:64] FLAG: --pods-per-core="0" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615790 2571 flags.go:64] FLAG: --port="10250" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615793 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615797 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a5aa3bb4c6196bc" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615800 2571 flags.go:64] FLAG: --qos-reserved="" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615803 2571 flags.go:64] FLAG: --read-only-port="10255" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615806 2571 flags.go:64] FLAG: --register-node="true" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615809 2571 flags.go:64] FLAG: --register-with-taints="" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615812 2571 flags.go:64] FLAG: --registry-burst="10" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615815 2571 flags.go:64] FLAG: --registry-qps="5" Aug 04 00:02:40.622505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615817 2571 flags.go:64] FLAG: --reserved-cpus="" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615820 2571 flags.go:64] FLAG: --reserved-memory="" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615824 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615827 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615834 2571 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615839 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615842 2571 flags.go:64] FLAG: --runonce="false" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615845 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615848 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615851 2571 flags.go:64] FLAG: --seccomp-default="false" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615854 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615857 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615860 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615863 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615866 2571 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615869 2571 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615872 2571 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615874 2571 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615877 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615880 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615883 2571 flags.go:64] FLAG: --system-cgroups="" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615886 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615891 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615894 2571 flags.go:64] FLAG: --tls-cert-file="" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615897 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 00:02:40.623119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615901 2571 flags.go:64] FLAG: --tls-min-version="" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615904 2571 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615907 2571 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615910 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615913 2571 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615915 2571 flags.go:64] FLAG: --v="2" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615919 2571 flags.go:64] FLAG: --version="false" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615923 2571 flags.go:64] FLAG: --vmodule="" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615927 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.615932 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616061 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616065 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616069 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616072 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616074 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616077 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616080 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616082 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616085 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616087 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616090 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616093 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:40.623777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616096 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616100 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616104 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616107 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616109 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616112 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616114 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616117 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616120 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616123 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616125 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616128 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616131 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616133 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616136 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616138 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616141 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616143 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616146 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616149 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:40.624320 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616152 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616154 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616158 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616161 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616164 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616166 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616169 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616172 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616174 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616177 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616179 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616181 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616184 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616187 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616189 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616191 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616194 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616196 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616199 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616201 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:40.624824 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616203 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616206 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616209 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616211 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616214 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616216 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616219 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616222 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616224 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616227 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616229 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616233 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616236 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616238 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616244 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616247 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616251 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616253 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616256 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:40.625327 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616260 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616263 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616266 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616269 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616272 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616274 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616277 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616279 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616282 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616285 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616287 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616290 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616292 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616295 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616297 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616300 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616302 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616305 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616307 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616311 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:40.625851 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616313 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616316 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616318 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616321 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616324 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616327 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616330 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.616333 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.617190 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.623952 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.623967 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.623986 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.623992 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624046 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:40.626338 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624051 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624054 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624057 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624060 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624063 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624067 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624071 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624074 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624078 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624081 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624083 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624086 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624090 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624093 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624096 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624098 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624101 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624104 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624107 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:40.626722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624110 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624113 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624116 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624118 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624122 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624124 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624127 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624129 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624132 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624136 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624140 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624142 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624145 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624148 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624151 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624153 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624156 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624159 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624161 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:40.627173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624165 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624169 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624172 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624175 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624179 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624183 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624186 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624188 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624191 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624194 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624197 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624199 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624202 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624204 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624207 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624209 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624212 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624214 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624217 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624219 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:40.627660 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624222 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624225 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624228 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624231 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624234 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624237 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624239 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624242 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624244 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624247 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624250 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624252 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624255 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624257 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624260 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624263 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624265 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624268 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624270 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624273 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:40.628185 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624275 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624278 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624280 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624283 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624286 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624288 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624291 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624293 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624296 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624299 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624301 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624304 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624306 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624309 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624312 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624315 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624318 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624321 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624324 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:40.628688 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624327 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.624332 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624439 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624445 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624448 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624451 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624454 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624476 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624481 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624484 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624487 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624489 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624492 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624495 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624497 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 00:02:40.629173 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624500 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624503 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624505 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624507 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624510 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624513 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624515 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624517 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624520 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624523 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624525 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624528 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624531 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624534 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624536 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624540 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624543 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624545 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624548 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624550 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 00:02:40.629581 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624553 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624556 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624559 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624563 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624566 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624569 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624571 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624574 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624577 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624579 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624583 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624586 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624589 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624592 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624595 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624598 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624601 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624603 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624606 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624609 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 00:02:40.630079 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624612 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624614 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624617 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624620 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624623 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624625 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624628 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624630 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624635 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624638 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624640 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624643 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624646 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624649 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624651 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624655 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624657 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624660 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624663 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624665 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 00:02:40.630584 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624668 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624670 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624672 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624675 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624677 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624680 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624682 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624685 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624687 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624690 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624692 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624695 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624697 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624699 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624702 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624704 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624707 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624710 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624712 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 00:02:40.631085 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624715 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624717 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624721 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624723 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624726 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624728 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:40.624731 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.624736 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.624851 2571 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.628628 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.629868 2571 server.go:1019] "Starting client certificate rotation" Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.629966 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 00:02:40.631571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.629996 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 00:02:40.634308 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.634288 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 00:02:40.636301 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.636284 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 00:02:40.648057 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.648033 2571 log.go:25] "Validated CRI v1 runtime API" Aug 04 00:02:40.650292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.650275 2571 log.go:25] "Validated CRI v1 image API" Aug 04 00:02:40.651106 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.651091 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 00:02:40.654504 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.654453 2571 fs.go:135] Filesystem UUIDs: map[5a79122b-fa0d-4d08-ba50-ff099ab26b57:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e9bead66-a774-4193-93e9-aeb2e7288d11:/dev/nvme0n1p3] Aug 04 00:02:40.654608 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.654503 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 00:02:40.660610 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.660484 2571 manager.go:216] Machine: {Timestamp:2026-08-04 00:02:40.658240677 +0000 UTC m=+0.351913843 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098844 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff5f4984d1ed25386f9989e6d861c SystemUUID:ec2ff5f4-984d-1ed2-5386-f9989e6d861c BootID:8f083b15-c718-48e4-aed3-c00a68143673 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3b:96:3b:37:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3b:96:3b:37:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:cc:45:fd:0f:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 00:02:40.660610 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.660597 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 00:02:40.660784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.660684 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 00:02:40.662428 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.662412 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 00:02:40.664841 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.664810 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 00:02:40.664982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.664850 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 00:02:40.665888 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.665878 2571 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 00:02:40.665930 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.665891 2571 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 00:02:40.665930 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.665898 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 00:02:40.665930 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.665909 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 00:02:40.666727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.666717 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 00:02:40.670775 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670763 2571 state_mem.go:36] "Initialized new in-memory state store" Aug 04 00:02:40.670883 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670874 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 00:02:40.670915 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670909 2571 kubelet.go:486] "Attempting to sync node with API server" Aug 04 00:02:40.670943 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670918 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 00:02:40.670943 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670933 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 00:02:40.670943 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670942 2571 kubelet.go:398] "Adding apiserver pod source" Aug 04 00:02:40.671047 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.670953 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 00:02:40.671301 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.671290 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 00:02:40.671332 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.671317 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 00:02:40.675099 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.675072 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 00:02:40.676903 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.676887 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 00:02:40.676985 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.676918 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 00:02:40.678251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678234 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 00:02:40.678251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678252 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678258 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678264 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678273 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678279 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678288 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678294 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678301 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678307 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678316 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 00:02:40.678351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.678322 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 00:02:40.679083 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.679067 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 00:02:40.679083 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.679079 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 00:02:40.681439 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.681419 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 00:02:40.682268 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.682251 2571 server.go:1268] "Started kubelet" Aug 04 00:02:40.682337 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.682281 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 00:02:40.682387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.682319 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 00:02:40.682387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.682339 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 00:02:40.682387 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.682374 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 00:02:40.682548 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.682402 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 00:02:40.683112 ip-10-0-134-15 systemd[1]: Started Kubernetes Kubelet. Aug 04 00:02:40.683758 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.683626 2571 server.go:310] "Adding debug handlers to kubelet server" Aug 04 00:02:40.683758 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.683729 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 00:02:40.689163 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.689147 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 00:02:40.689451 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.689435 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 00:02:40.689532 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.689453 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 00:02:40.689580 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.689525 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:40.690142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690123 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 00:02:40.690142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690126 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 00:02:40.690295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690156 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 00:02:40.690295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690240 2571 reconstruct.go:97] "Volume reconstruction finished" Aug 04 00:02:40.690295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690245 2571 reconciler.go:29] "Reconciler: start to sync state" Aug 04 00:02:40.690432 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690346 2571 factory.go:55] Registering systemd factory Aug 04 00:02:40.690432 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690362 2571 factory.go:223] Registration of the systemd container factory successfully Aug 04 00:02:40.690585 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690569 2571 factory.go:153] Registering CRI-O factory Aug 04 00:02:40.690585 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690587 2571 factory.go:223] Registration of the crio container factory successfully Aug 04 00:02:40.690727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690631 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 00:02:40.690727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690661 2571 factory.go:103] Registering Raw factory Aug 04 00:02:40.690727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.690674 2571 manager.go:1196] Started watching for new ooms in manager Aug 04 00:02:40.690727 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.689800 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-15.ec2.internal.18c8713d60bb31b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-15.ec2.internal,UID:ip-10-0-134-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-15.ec2.internal,},FirstTimestamp:2026-08-04 00:02:40.682226097 +0000 UTC m=+0.375899264,LastTimestamp:2026-08-04 00:02:40.682226097 +0000 UTC m=+0.375899264,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-15.ec2.internal,}" Aug 04 00:02:40.691031 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.691018 2571 manager.go:349] Starting recovery of all containers Aug 04 00:02:40.693702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.693678 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rzslz" Aug 04 00:02:40.694884 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.694856 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 00:02:40.694980 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.694919 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 00:02:40.704213 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.704195 2571 manager.go:354] Recovery completed Aug 04 00:02:40.706741 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.706725 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rzslz" Aug 04 00:02:40.709171 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.709157 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:40.711057 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711042 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:40.711138 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711067 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:40.711138 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711079 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:40.711676 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711662 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 00:02:40.711739 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711678 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 00:02:40.711739 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711696 2571 state_mem.go:36] "Initialized new in-memory state store" Aug 04 00:02:40.711739 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.711665 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:40.712416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.712400 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:40.712510 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.712430 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:40.712510 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.712444 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:40.712903 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.712883 2571 policy_none.go:49] "None policy: Start" Aug 04 00:02:40.712903 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.712906 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 00:02:40.713064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.712921 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 00:02:40.713142 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.713078 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-15.ec2.internal.18c8713d6273183b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-15.ec2.internal,UID:ip-10-0-134-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-15.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-15.ec2.internal,},FirstTimestamp:2026-08-04 00:02:40.711055419 +0000 UTC m=+0.404728588,LastTimestamp:2026-08-04 00:02:40.711055419 +0000 UTC m=+0.404728588,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-15.ec2.internal,}" Aug 04 00:02:40.714017 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.714004 2571 policy_none.go:47] "Start" Aug 04 00:02:40.765117 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.764987 2571 manager.go:337] "Starting Device Plugin manager" Aug 04 00:02:40.765248 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.765156 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 00:02:40.765248 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.765172 2571 server.go:87] "Starting device plugin registration server" Aug 04 00:02:40.765395 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.765384 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 00:02:40.765442 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.765396 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 00:02:40.765523 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.765499 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 00:02:40.765571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.765560 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 00:02:40.765571 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:40.765568 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 00:02:40.766052 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.766035 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 00:02:40.766117 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:40.766082 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.368925 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.368893 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 00:02:42.370133 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.370092 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 00:02:42.370238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.370145 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 00:02:42.370238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.370169 2571 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 00:02:42.370238 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.370204 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 00:02:42.372324 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.372308 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:42.389135 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.389053 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.389971 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.389954 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.390063 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.389988 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.390063 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.390006 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.390063 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.390033 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.401293 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.401269 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.401352 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.401301 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-15.ec2.internal\": node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.429196 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.429162 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.470980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.470947 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal"] Aug 04 00:02:42.471145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.471022 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.471922 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.471905 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.472040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.471937 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.472040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.471951 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.473361 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.473346 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.473504 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.473488 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.473559 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.473532 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.474143 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.474129 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.474217 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.474161 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.474217 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.474171 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.474217 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.474129 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.474302 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.474231 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.474302 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.474249 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.476130 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.476116 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.476176 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.476149 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 00:02:42.476822 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.476807 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientMemory" Aug 04 00:02:42.476918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.476833 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 00:02:42.476918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.476848 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeHasSufficientPID" Aug 04 00:02:42.489641 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.489620 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-15.ec2.internal\" not found" node="ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.493370 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.493355 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-15.ec2.internal\" not found" node="ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.514146 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.514089 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96edcf5a52319a9f781fc5c1509e4df3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal\" (UID: \"96edcf5a52319a9f781fc5c1509e4df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.514146 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.514147 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96edcf5a52319a9f781fc5c1509e4df3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal\" (UID: \"96edcf5a52319a9f781fc5c1509e4df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.514311 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.514173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b91cd527bb73e98102acd9e9c144cdda-config\") pod \"kube-apiserver-proxy-ip-10-0-134-15.ec2.internal\" (UID: \"b91cd527bb73e98102acd9e9c144cdda\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.529490 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.529468 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.614734 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.614701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96edcf5a52319a9f781fc5c1509e4df3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal\" (UID: \"96edcf5a52319a9f781fc5c1509e4df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.614861 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.614736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96edcf5a52319a9f781fc5c1509e4df3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal\" (UID: \"96edcf5a52319a9f781fc5c1509e4df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.614861 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.614754 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b91cd527bb73e98102acd9e9c144cdda-config\") pod \"kube-apiserver-proxy-ip-10-0-134-15.ec2.internal\" (UID: \"b91cd527bb73e98102acd9e9c144cdda\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.614861 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.614797 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b91cd527bb73e98102acd9e9c144cdda-config\") pod \"kube-apiserver-proxy-ip-10-0-134-15.ec2.internal\" (UID: \"b91cd527bb73e98102acd9e9c144cdda\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.614861 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.614802 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96edcf5a52319a9f781fc5c1509e4df3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal\" (UID: \"96edcf5a52319a9f781fc5c1509e4df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.614861 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.614816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96edcf5a52319a9f781fc5c1509e4df3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal\" (UID: \"96edcf5a52319a9f781fc5c1509e4df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.629857 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.629816 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.730711 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.730639 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.792859 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.792829 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.795991 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:42.795961 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" Aug 04 00:02:42.831134 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.831093 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:42.931628 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:42.931581 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.032215 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:43.032130 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.132633 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:43.132606 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.151789 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.151762 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 00:02:43.151934 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.151900 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 00:02:43.194721 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.194688 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:43.233051 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:43.233023 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.234177 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.234121 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-02 23:57:42 +0000 UTC" deadline="2028-02-23 15:24:55.065034179 +0000 UTC" Aug 04 00:02:43.234177 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.234144 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13647h22m11.830893163s" Aug 04 00:02:43.296145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.296055 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:43.334191 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:43.334161 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.336038 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:43.336002 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91cd527bb73e98102acd9e9c144cdda.slice/crio-a52cb6dd11bf3a05ca0720d81f68a7b78a9c6d018f12bdd2047c0b79694fcaa5 WatchSource:0}: Error finding container a52cb6dd11bf3a05ca0720d81f68a7b78a9c6d018f12bdd2047c0b79694fcaa5: Status 404 returned error can't find the container with id a52cb6dd11bf3a05ca0720d81f68a7b78a9c6d018f12bdd2047c0b79694fcaa5 Aug 04 00:02:43.336248 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:43.336226 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96edcf5a52319a9f781fc5c1509e4df3.slice/crio-b744cfb57e5298b9c2193e4d635242f9656ba0e99c69a6af74fe0c7d6e39c831 WatchSource:0}: Error finding container b744cfb57e5298b9c2193e4d635242f9656ba0e99c69a6af74fe0c7d6e39c831: Status 404 returned error can't find the container with id b744cfb57e5298b9c2193e4d635242f9656ba0e99c69a6af74fe0c7d6e39c831 Aug 04 00:02:43.339630 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.339606 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:02:43.373772 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.373713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" event={"ID":"96edcf5a52319a9f781fc5c1509e4df3","Type":"ContainerStarted","Data":"b744cfb57e5298b9c2193e4d635242f9656ba0e99c69a6af74fe0c7d6e39c831"} Aug 04 00:02:43.374738 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.374711 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" event={"ID":"b91cd527bb73e98102acd9e9c144cdda","Type":"ContainerStarted","Data":"a52cb6dd11bf3a05ca0720d81f68a7b78a9c6d018f12bdd2047c0b79694fcaa5"} Aug 04 00:02:43.435190 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:43.435157 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.535677 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:43.535649 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-15.ec2.internal\" not found" Aug 04 00:02:43.548988 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.548937 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:43.612690 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.612645 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" Aug 04 00:02:43.622609 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.622584 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 00:02:43.623469 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.623456 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" Aug 04 00:02:43.631066 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:43.631049 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 00:02:44.194781 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.194745 2571 apiserver.go:52] "Watching apiserver" Aug 04 00:02:44.201933 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.201908 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 00:02:44.202378 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.202351 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-sbgfn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b","openshift-cluster-node-tuning-operator/tuned-xhg89","openshift-dns/node-resolver-kx9jg","openshift-image-registry/node-ca-tpvjb","openshift-multus/multus-additional-cni-plugins-46gf6","openshift-network-diagnostics/network-check-target-2tbpf","openshift-network-operator/iptables-alerter-zvx57","kube-system/konnectivity-agent-s66z8","kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal","openshift-multus/multus-6t2wv","openshift-multus/network-metrics-daemon-9wzbx","openshift-ovn-kubernetes/ovnkube-node-v42gn"] Aug 04 00:02:44.204061 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.204043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:44.204159 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.204124 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:44.205407 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.205369 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.205511 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.205447 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:44.206486 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.206468 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.208300 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.208281 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.208543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.208422 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.208543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.208424 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.208738 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.208721 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vs7sl\"" Aug 04 00:02:44.209370 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.209348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.210088 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.210066 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.210202 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.210178 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqbpk\"" Aug 04 00:02:44.210710 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.210687 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.210806 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.210789 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.211405 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.211376 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.211668 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.211592 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-76cpt\"" Aug 04 00:02:44.211668 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.211617 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 00:02:44.211817 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.211621 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.212157 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.212139 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.213230 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.213211 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 00:02:44.213832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.213710 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2sdw4\"" Aug 04 00:02:44.213832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.213755 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.213832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.213765 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 00:02:44.213832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.213710 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 00:02:44.213832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.213819 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 00:02:44.214614 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.214593 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.217628 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.215184 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 00:02:44.217628 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.215581 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.217751 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.217708 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:44.217751 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.215616 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-plwr8\"" Aug 04 00:02:44.217847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.215860 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 00:02:44.218312 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.218295 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.220664 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.220647 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.220952 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.220934 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.221127 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.221090 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 00:02:44.221235 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.221139 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shf2g\"" Aug 04 00:02:44.221381 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.221362 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.222160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222136 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.222541 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222517 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-var-lib-kubelet\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.222636 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222555 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46vz\" (UniqueName: \"kubernetes.io/projected/3ba07224-77d2-4138-873b-4278e1b0cbb5-kube-api-access-m46vz\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.222636 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222581 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-os-release\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.222748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222645 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 00:02:44.222748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.222748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222678 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ae9a103-10de-4451-bb24-97976795b3c5-kubelet-config\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.222748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-lib-modules\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.222748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222718 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmpf\" (UniqueName: \"kubernetes.io/projected/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-kube-api-access-kdmpf\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.222748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222737 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-hosts-file\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f95614fb-0969-4b83-9db8-7bbc91885d23-host\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222817 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222870 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysctl-conf\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222928 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.222981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cni-binary-copy\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.223030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysconfig\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-host\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223064 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-tmp\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223086 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ae9a103-10de-4451-bb24-97976795b3c5-dbus\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223126 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysctl-d\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223148 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-tuned\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-tmp-dir\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223195 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbvl\" (UniqueName: \"kubernetes.io/projected/f95614fb-0969-4b83-9db8-7bbc91885d23-kube-api-access-7bbvl\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223202 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-run\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223261 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cnibin\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223284 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnc4\" (UniqueName: \"kubernetes.io/projected/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-kube-api-access-hcnc4\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223303 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-systemd\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b85h\" (UniqueName: \"kubernetes.io/projected/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-kube-api-access-9b85h\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.223444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223437 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-system-cni-dir\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223477 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/116551c0-6d82-4e9e-be71-7a11d2ecf733-konnectivity-ca\") pod \"konnectivity-agent-s66z8\" (UID: \"116551c0-6d82-4e9e-be71-7a11d2ecf733\") " pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223490 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223491 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-kubernetes\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f95614fb-0969-4b83-9db8-7bbc91885d23-serviceca\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223591 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/116551c0-6d82-4e9e-be71-7a11d2ecf733-agent-certs\") pod \"konnectivity-agent-s66z8\" (UID: \"116551c0-6d82-4e9e-be71-7a11d2ecf733\") " pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223614 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-modprobe-d\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-sys\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223690 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lgxcd\"" Aug 04 00:02:44.224019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.223730 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 00:02:44.224413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.224063 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 00:02:44.224413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.224158 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-scmlx\"" Aug 04 00:02:44.224413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.224206 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:02:44.224413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.224383 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 00:02:44.225085 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.225069 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.226194 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.226176 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 00:02:44.226872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.226856 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 00:02:44.226872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.226866 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zw9fr\"" Aug 04 00:02:44.243163 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.243144 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n9h6f" Aug 04 00:02:44.252450 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.252431 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n9h6f" Aug 04 00:02:44.313930 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.313896 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 00:02:44.324457 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-cnibin\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.324588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-conf-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.324588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-etc-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.324588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324513 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.324588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324540 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.324588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.324845 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysconfig\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.324845 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-tmp\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.324845 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324766 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.324845 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324780 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-netns\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.324845 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324828 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-sys-fs\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.325062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ae9a103-10de-4451-bb24-97976795b3c5-dbus\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.325062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbjw\" (UniqueName: \"kubernetes.io/projected/051a04fe-978e-48ea-878f-71bf7c22419e-kube-api-access-spbjw\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.325062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.325062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324783 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysconfig\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.325062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.324942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-cni-multus\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325063 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325025 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-slash\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cnibin\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325147 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ae9a103-10de-4451-bb24-97976795b3c5-dbus\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cnibin\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325188 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b85h\" (UniqueName: \"kubernetes.io/projected/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-kube-api-access-9b85h\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.325295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325238 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-device-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325282 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051a04fe-978e-48ea-878f-71bf7c22419e-multus-daemon-config\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325449 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-system-cni-dir\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/116551c0-6d82-4e9e-be71-7a11d2ecf733-konnectivity-ca\") pod \"konnectivity-agent-s66z8\" (UID: \"116551c0-6d82-4e9e-be71-7a11d2ecf733\") " pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325501 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-system-cni-dir\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325513 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f95614fb-0969-4b83-9db8-7bbc91885d23-serviceca\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325535 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-cni-bin\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.325705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.325580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-node-log\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.326081 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/116551c0-6d82-4e9e-be71-7a11d2ecf733-agent-certs\") pod \"konnectivity-agent-s66z8\" (UID: \"116551c0-6d82-4e9e-be71-7a11d2ecf733\") " pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.326160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-sys\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.326214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326183 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-os-release\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.326269 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326220 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-socket-dir-parent\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.326269 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-registration-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.326362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326291 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72hj\" (UniqueName: \"kubernetes.io/projected/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-kube-api-access-q72hj\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.326362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326322 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-var-lib-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.326362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326350 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovn-node-metrics-cert\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.326507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326387 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46vz\" (UniqueName: \"kubernetes.io/projected/3ba07224-77d2-4138-873b-4278e1b0cbb5-kube-api-access-m46vz\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.326507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-os-release\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.326507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326457 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.326507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326490 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ae9a103-10de-4451-bb24-97976795b3c5-kubelet-config\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.326689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326517 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-lib-modules\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.326689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.326604 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f95614fb-0969-4b83-9db8-7bbc91885d23-host\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f95614fb-0969-4b83-9db8-7bbc91885d23-serviceca\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327435 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-sys\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ae9a103-10de-4451-bb24-97976795b3c5-kubelet-config\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327783 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-lib-modules\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327791 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-os-release\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f95614fb-0969-4b83-9db8-7bbc91885d23-host\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327924 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-cni-netd\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.327993 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328028 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysctl-conf\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051a04fe-978e-48ea-878f-71bf7c22419e-cni-binary-copy\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328058 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-multus-certs\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-env-overrides\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328174 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cni-binary-copy\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328173 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/116551c0-6d82-4e9e-be71-7a11d2ecf733-konnectivity-ca\") pod \"konnectivity-agent-s66z8\" (UID: \"116551c0-6d82-4e9e-be71-7a11d2ecf733\") " pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328231 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-host\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.329284 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.328278 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328432 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysctl-conf\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328481 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-k8s-cni-cncf-io\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328517 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-run-netns\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328545 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-log-socket\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328577 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-cni-bin\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328641 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovnkube-script-lib\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328671 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-cni-binary-copy\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bft\" (UniqueName: \"kubernetes.io/projected/7357fd0a-8ae8-4dca-9466-78f85c4efa53-kube-api-access-f7bft\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328744 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysctl-d\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328768 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-host\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.328783 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:44.828740656 +0000 UTC m=+2.999773960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-tuned\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328860 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-tmp-dir\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328875 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-sysctl-d\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.328892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbvl\" (UniqueName: \"kubernetes.io/projected/f95614fb-0969-4b83-9db8-7bbc91885d23-kube-api-access-7bbvl\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.330254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329004 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-etc-kubernetes\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329165 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-systemd-units\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329197 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-run-ovn-kubernetes\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329284 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-tmp-dir\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329324 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-run\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329358 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-system-cni-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-kubelet\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329424 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329462 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnc4\" (UniqueName: \"kubernetes.io/projected/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-kube-api-access-hcnc4\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329561 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-systemd\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329582 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/116551c0-6d82-4e9e-be71-7a11d2ecf733-agent-certs\") pod \"konnectivity-agent-s66z8\" (UID: \"116551c0-6d82-4e9e-be71-7a11d2ecf733\") " pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329597 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-socket-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329660 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-run\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329689 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-systemd\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.329717 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329711 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-ovn\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovnkube-config\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.329874 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:44.829856351 +0000 UTC m=+3.000889647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.329955 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-kubernetes\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330001 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-cni-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330070 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-kubernetes\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-hostroot\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-kubelet\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-systemd\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330237 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-modprobe-d\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-var-lib-kubelet\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c029686e-4ca6-4da0-8682-98a0bc020e86-iptables-alerter-script\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbcg\" (UniqueName: \"kubernetes.io/projected/c029686e-4ca6-4da0-8682-98a0bc020e86-kube-api-access-lwbcg\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330370 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmpf\" (UniqueName: \"kubernetes.io/projected/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-kube-api-access-kdmpf\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330382 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-var-lib-kubelet\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-hosts-file\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330386 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-modprobe-d\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.331766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330437 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c029686e-4ca6-4da0-8682-98a0bc020e86-host-slash\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.332510 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.330601 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-hosts-file\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.332510 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.332008 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-tmp\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.332510 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.332262 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-etc-tuned\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.338279 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.338257 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:44.338391 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.338284 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:44.338391 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.338300 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:44.338391 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.338357 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:44.838342446 +0000 UTC m=+3.009375737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:44.340650 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.340475 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b85h\" (UniqueName: \"kubernetes.io/projected/539e58ad-eeb7-4a35-8d3f-af0cebbbe186-kube-api-access-9b85h\") pod \"node-resolver-kx9jg\" (UID: \"539e58ad-eeb7-4a35-8d3f-af0cebbbe186\") " pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.340650 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.340573 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbvl\" (UniqueName: \"kubernetes.io/projected/f95614fb-0969-4b83-9db8-7bbc91885d23-kube-api-access-7bbvl\") pod \"node-ca-tpvjb\" (UID: \"f95614fb-0969-4b83-9db8-7bbc91885d23\") " pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.341321 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.341273 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46vz\" (UniqueName: \"kubernetes.io/projected/3ba07224-77d2-4138-873b-4278e1b0cbb5-kube-api-access-m46vz\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.341461 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.341441 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnc4\" (UniqueName: \"kubernetes.io/projected/e000d8a9-3ea5-4106-96cc-2a8b36c1a680-kube-api-access-hcnc4\") pod \"multus-additional-cni-plugins-46gf6\" (UID: \"e000d8a9-3ea5-4106-96cc-2a8b36c1a680\") " pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.341593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.341574 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmpf\" (UniqueName: \"kubernetes.io/projected/8faa5588-36ea-40c7-b1b3-09cb04b05d0a-kube-api-access-kdmpf\") pod \"tuned-xhg89\" (UID: \"8faa5588-36ea-40c7-b1b3-09cb04b05d0a\") " pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.431830 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c029686e-4ca6-4da0-8682-98a0bc020e86-iptables-alerter-script\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.431830 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431835 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbcg\" (UniqueName: \"kubernetes.io/projected/c029686e-4ca6-4da0-8682-98a0bc020e86-kube-api-access-lwbcg\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c029686e-4ca6-4da0-8682-98a0bc020e86-host-slash\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-cnibin\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431910 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-conf-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-etc-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.431979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-netns\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-sys-fs\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432052 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbjw\" (UniqueName: \"kubernetes.io/projected/051a04fe-978e-48ea-878f-71bf7c22419e-kube-api-access-spbjw\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432076 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432117 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-cni-multus\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-slash\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-device-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432205 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c029686e-4ca6-4da0-8682-98a0bc020e86-host-slash\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051a04fe-978e-48ea-878f-71bf7c22419e-multus-daemon-config\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432264 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-cni-bin\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432280 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-cnibin\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432289 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-node-log\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-os-release\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432347 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-socket-dir-parent\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432376 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-registration-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432387 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72hj\" (UniqueName: \"kubernetes.io/projected/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-kube-api-access-q72hj\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432418 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-netns\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.432475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432444 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c029686e-4ca6-4da0-8682-98a0bc020e86-iptables-alerter-script\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-var-lib-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432493 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-os-release\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432318 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-conf-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432507 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-device-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovn-node-metrics-cert\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432546 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-socket-dir-parent\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432349 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-etc-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432567 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-cni-netd\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-registration-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432614 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-cni-bin\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432625 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051a04fe-978e-48ea-878f-71bf7c22419e-cni-binary-copy\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432649 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-cni-multus\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-multus-certs\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432684 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-node-log\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.433335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-env-overrides\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432728 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-sys-fs\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-k8s-cni-cncf-io\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432735 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-var-lib-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432759 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-run-netns\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-log-socket\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432783 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-slash\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432824 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-multus-certs\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432829 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-run-k8s-cni-cncf-io\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432853 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-run-netns\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-cni-bin\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432909 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-cni-netd\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432898 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-log-socket\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432950 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovnkube-script-lib\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.432979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bft\" (UniqueName: \"kubernetes.io/projected/7357fd0a-8ae8-4dca-9466-78f85c4efa53-kube-api-access-f7bft\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433012 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-etc-kubernetes\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433038 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-systemd-units\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-run-ovn-kubernetes\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-cni-bin\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433128 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-run-ovn-kubernetes\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433193 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-etc-kubernetes\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433262 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-env-overrides\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433328 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-systemd-units\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-system-cni-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433398 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-kubelet\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433471 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-socket-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433485 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-system-cni-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-ovn\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433523 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-host-var-lib-kubelet\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433527 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovnkube-config\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433565 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-ovn\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433592 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-cni-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433604 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-socket-dir\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433497 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-openvswitch\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433649 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-hostroot\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.434919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433651 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovnkube-script-lib\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433670 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-multus-cni-dir\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433619 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051a04fe-978e-48ea-878f-71bf7c22419e-hostroot\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-kubelet\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-systemd\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433772 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051a04fe-978e-48ea-878f-71bf7c22419e-multus-daemon-config\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433774 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-host-kubelet\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.433804 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7357fd0a-8ae8-4dca-9466-78f85c4efa53-run-systemd\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.434068 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovnkube-config\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.434170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051a04fe-978e-48ea-878f-71bf7c22419e-cni-binary-copy\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.435632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.435457 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7357fd0a-8ae8-4dca-9466-78f85c4efa53-ovn-node-metrics-cert\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.443881 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.443855 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbcg\" (UniqueName: \"kubernetes.io/projected/c029686e-4ca6-4da0-8682-98a0bc020e86-kube-api-access-lwbcg\") pod \"iptables-alerter-zvx57\" (UID: \"c029686e-4ca6-4da0-8682-98a0bc020e86\") " pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.444357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.444331 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbjw\" (UniqueName: \"kubernetes.io/projected/051a04fe-978e-48ea-878f-71bf7c22419e-kube-api-access-spbjw\") pod \"multus-6t2wv\" (UID: \"051a04fe-978e-48ea-878f-71bf7c22419e\") " pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.444357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.444348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72hj\" (UniqueName: \"kubernetes.io/projected/4f4cf4fc-667a-46c8-8828-a509aafaa4ae-kube-api-access-q72hj\") pod \"aws-ebs-csi-driver-node-xjk4b\" (UID: \"4f4cf4fc-667a-46c8-8828-a509aafaa4ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.444544 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.444450 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bft\" (UniqueName: \"kubernetes.io/projected/7357fd0a-8ae8-4dca-9466-78f85c4efa53-kube-api-access-f7bft\") pod \"ovnkube-node-v42gn\" (UID: \"7357fd0a-8ae8-4dca-9466-78f85c4efa53\") " pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.449343 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.449275 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 00:02:44.526141 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.526096 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhg89" Aug 04 00:02:44.531412 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.531394 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kx9jg" Aug 04 00:02:44.534215 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.534193 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faa5588_36ea_40c7_b1b3_09cb04b05d0a.slice/crio-750e8ee752aceac2c25f545ad65bb45363d400fbe2dbba5a866f7a09487afbb9 WatchSource:0}: Error finding container 750e8ee752aceac2c25f545ad65bb45363d400fbe2dbba5a866f7a09487afbb9: Status 404 returned error can't find the container with id 750e8ee752aceac2c25f545ad65bb45363d400fbe2dbba5a866f7a09487afbb9 Aug 04 00:02:44.537952 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.537924 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539e58ad_eeb7_4a35_8d3f_af0cebbbe186.slice/crio-1948c7f94c90081dac032929157b76c272727744e33ff245fb4481ea81257452 WatchSource:0}: Error finding container 1948c7f94c90081dac032929157b76c272727744e33ff245fb4481ea81257452: Status 404 returned error can't find the container with id 1948c7f94c90081dac032929157b76c272727744e33ff245fb4481ea81257452 Aug 04 00:02:44.538908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.538881 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tpvjb" Aug 04 00:02:44.545141 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.545121 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-46gf6" Aug 04 00:02:44.548386 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.548357 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95614fb_0969_4b83_9db8_7bbc91885d23.slice/crio-9cf5c7e34713d9344a2efafd21198ccca2edf1cc24cab20ab284286909524e47 WatchSource:0}: Error finding container 9cf5c7e34713d9344a2efafd21198ccca2edf1cc24cab20ab284286909524e47: Status 404 returned error can't find the container with id 9cf5c7e34713d9344a2efafd21198ccca2edf1cc24cab20ab284286909524e47 Aug 04 00:02:44.552573 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.552552 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:02:44.552812 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.552792 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode000d8a9_3ea5_4106_96cc_2a8b36c1a680.slice/crio-66bf4dac16190e544332e1c6c2ee3eacb8a0a145c9801b914a5388ae08db8501 WatchSource:0}: Error finding container 66bf4dac16190e544332e1c6c2ee3eacb8a0a145c9801b914a5388ae08db8501: Status 404 returned error can't find the container with id 66bf4dac16190e544332e1c6c2ee3eacb8a0a145c9801b914a5388ae08db8501 Aug 04 00:02:44.559744 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.559723 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" Aug 04 00:02:44.560343 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.560324 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116551c0_6d82_4e9e_be71_7a11d2ecf733.slice/crio-fc49519a5df19a926b1b9a715f3af451c16018acb8ad4b82d1593fec566ed105 WatchSource:0}: Error finding container fc49519a5df19a926b1b9a715f3af451c16018acb8ad4b82d1593fec566ed105: Status 404 returned error can't find the container with id fc49519a5df19a926b1b9a715f3af451c16018acb8ad4b82d1593fec566ed105 Aug 04 00:02:44.566862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.566845 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:02:44.567033 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.567012 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4cf4fc_667a_46c8_8828_a509aafaa4ae.slice/crio-739887450ba722bf59324244260a3b5c00a9a44b0cf13d18a628b6429d66caba WatchSource:0}: Error finding container 739887450ba722bf59324244260a3b5c00a9a44b0cf13d18a628b6429d66caba: Status 404 returned error can't find the container with id 739887450ba722bf59324244260a3b5c00a9a44b0cf13d18a628b6429d66caba Aug 04 00:02:44.573354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.573334 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zvx57" Aug 04 00:02:44.579411 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.579341 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6t2wv" Aug 04 00:02:44.588806 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:02:44.588781 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051a04fe_978e_48ea_878f_71bf7c22419e.slice/crio-6327898c2026705c96ad66edabf8704275676ea1e6804c0f0ebc9b093c3f9d70 WatchSource:0}: Error finding container 6327898c2026705c96ad66edabf8704275676ea1e6804c0f0ebc9b093c3f9d70: Status 404 returned error can't find the container with id 6327898c2026705c96ad66edabf8704275676ea1e6804c0f0ebc9b093c3f9d70 Aug 04 00:02:44.836658 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.836476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:44.836824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.836670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:44.836824 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.836639 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:44.836824 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.836761 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:44.836824 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.836791 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:45.836771701 +0000 UTC m=+4.007804985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:44.836824 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.836811 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:45.836801721 +0000 UTC m=+4.007835008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:44.937258 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:44.937226 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:44.937448 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.937384 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:44.937448 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.937407 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:44.937448 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.937420 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:44.937613 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:44.937491 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:45.937474978 +0000 UTC m=+4.108508276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:45.255282 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.255214 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:57:44 +0000 UTC" deadline="2028-04-02 12:11:57.163254062 +0000 UTC" Aug 04 00:02:45.255282 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.255278 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14580h9m11.907980166s" Aug 04 00:02:45.370594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.370563 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:45.370752 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.370691 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:45.371353 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.371330 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:45.371480 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.371420 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:45.380789 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.380761 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s66z8" event={"ID":"116551c0-6d82-4e9e-be71-7a11d2ecf733","Type":"ContainerStarted","Data":"fc49519a5df19a926b1b9a715f3af451c16018acb8ad4b82d1593fec566ed105"} Aug 04 00:02:45.383189 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.383164 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerStarted","Data":"66bf4dac16190e544332e1c6c2ee3eacb8a0a145c9801b914a5388ae08db8501"} Aug 04 00:02:45.385720 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.385688 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tpvjb" event={"ID":"f95614fb-0969-4b83-9db8-7bbc91885d23","Type":"ContainerStarted","Data":"9cf5c7e34713d9344a2efafd21198ccca2edf1cc24cab20ab284286909524e47"} Aug 04 00:02:45.388292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.388266 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"de2c1228057789af13e5f4742fcb7f6b7e41e4f3f741768d60a0bd4e03521c7b"} Aug 04 00:02:45.396890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.396807 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kx9jg" event={"ID":"539e58ad-eeb7-4a35-8d3f-af0cebbbe186","Type":"ContainerStarted","Data":"1948c7f94c90081dac032929157b76c272727744e33ff245fb4481ea81257452"} Aug 04 00:02:45.403028 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.402976 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhg89" event={"ID":"8faa5588-36ea-40c7-b1b3-09cb04b05d0a","Type":"ContainerStarted","Data":"750e8ee752aceac2c25f545ad65bb45363d400fbe2dbba5a866f7a09487afbb9"} Aug 04 00:02:45.411659 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.411632 2571 generic.go:356] "Generic (PLEG): container finished" podID="96edcf5a52319a9f781fc5c1509e4df3" containerID="53aec6f6e7294bff7d851d7372d4f44a6cf04d44e89a6d2f0a3b651c0a8f5eae" exitCode=0 Aug 04 00:02:45.411776 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.411710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" event={"ID":"96edcf5a52319a9f781fc5c1509e4df3","Type":"ContainerDied","Data":"53aec6f6e7294bff7d851d7372d4f44a6cf04d44e89a6d2f0a3b651c0a8f5eae"} Aug 04 00:02:45.427502 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.427470 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6t2wv" event={"ID":"051a04fe-978e-48ea-878f-71bf7c22419e","Type":"ContainerStarted","Data":"6327898c2026705c96ad66edabf8704275676ea1e6804c0f0ebc9b093c3f9d70"} Aug 04 00:02:45.431341 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.431310 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvx57" event={"ID":"c029686e-4ca6-4da0-8682-98a0bc020e86","Type":"ContainerStarted","Data":"4f04f3580eb19845dcb93371e513fbc84975fe3a9ef88c694c077d5f32045ca6"} Aug 04 00:02:45.437965 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.437886 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" event={"ID":"4f4cf4fc-667a-46c8-8828-a509aafaa4ae","Type":"ContainerStarted","Data":"739887450ba722bf59324244260a3b5c00a9a44b0cf13d18a628b6429d66caba"} Aug 04 00:02:45.846440 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.846348 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:45.846440 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.846419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:45.846680 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.846638 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:45.846742 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.846703 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:47.846683771 +0000 UTC m=+6.017717069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:45.847161 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.847139 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:45.847254 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.847198 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:47.847182205 +0000 UTC m=+6.018215498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:45.947573 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:45.947536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:45.947751 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.947729 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:45.947826 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.947761 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:45.947826 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.947775 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:45.947932 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:45.947843 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:47.947822905 +0000 UTC m=+6.118856195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:46.256297 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:46.256205 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-02 23:57:44 +0000 UTC" deadline="2028-02-14 12:47:04.255998187 +0000 UTC" Aug 04 00:02:46.256297 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:46.256245 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13428h44m17.999757744s" Aug 04 00:02:46.375682 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:46.375650 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:46.375845 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:46.375775 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:46.469676 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:46.465407 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" event={"ID":"96edcf5a52319a9f781fc5c1509e4df3","Type":"ContainerStarted","Data":"346d142b1e8694a390796a8e39663b5fae4bbf8dffdc1132b2fcb590a78412e2"} Aug 04 00:02:47.371617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:47.371037 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:47.371617 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.371260 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:47.371617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:47.371288 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:47.371617 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.371414 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:47.864882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:47.864799 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:47.864882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:47.864866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:47.865173 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.865043 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:47.865173 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.865055 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:47.865173 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.865129 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.865094553 +0000 UTC m=+10.036127840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:47.865173 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.865151 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.865141487 +0000 UTC m=+10.036174775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:47.965508 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:47.965468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:47.966182 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.965699 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:47.966182 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.965728 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:47.966182 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.965741 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:47.966182 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:47.965801 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:51.965781318 +0000 UTC m=+10.136814608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:48.373633 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:48.370779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:48.373633 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:48.373227 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:49.371231 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:49.370676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:49.371231 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:49.370839 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:49.371231 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:49.370865 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:49.371231 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:49.370961 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:50.370438 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:50.370402 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:50.370876 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:50.370527 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:51.370669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:51.370634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:51.371120 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.370778 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:51.371120 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:51.370896 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:51.371120 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.370963 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:51.897693 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:51.897652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:51.897876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:51.897732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:51.897876 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.897811 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:51.897993 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.897887 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:59.897867051 +0000 UTC m=+18.068900347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:51.897993 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.897888 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:51.897993 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.897954 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:59.897934875 +0000 UTC m=+18.068968175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:51.998677 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:51.998562 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:51.998870 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.998777 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:02:51.998870 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.998804 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:02:51.998870 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.998817 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:51.999040 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:51.998890 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:02:59.998867282 +0000 UTC m=+18.169900580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:02:52.375172 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:52.375046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:52.375607 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:52.375183 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:53.370577 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:53.370538 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:53.370804 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:53.370697 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:53.371116 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:53.371086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:53.371205 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:53.371188 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:54.373982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:54.373956 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:54.374502 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:54.374080 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:55.370980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:55.370952 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:55.370980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:55.370985 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:55.371233 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:55.371049 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:55.371233 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:55.371122 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:56.371251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:56.371217 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:56.371730 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:56.371371 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:57.370947 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:57.370908 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:57.371139 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:57.370912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:57.371139 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:57.371055 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:57.371246 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:57.371150 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:58.370826 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:58.370786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:58.371347 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:58.370919 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:02:59.370867 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:59.370830 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:02:59.371377 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:59.370965 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:02:59.371377 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:59.371019 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:59.371377 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:59.371144 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:02:59.955722 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:59.955689 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:02:59.955890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:02:59.955737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:02:59.955890 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:59.955818 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:59.955890 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:59.955849 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:02:59.956017 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:59.955892 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:15.955871538 +0000 UTC m=+34.126904838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:02:59.956017 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:02:59.955909 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:15.955901836 +0000 UTC m=+34.126935120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:00.056298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:00.056258 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:00.056463 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:00.056405 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:03:00.056463 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:00.056424 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:03:00.056463 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:00.056435 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:00.056602 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:00.056500 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:16.056483462 +0000 UTC m=+34.227516969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:00.370641 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:00.370557 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:00.370784 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:00.370699 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:01.370835 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:01.370804 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:01.371340 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:01.370864 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:01.371340 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:01.370951 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:01.371340 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:01.371070 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:02.371860 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:02.371817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:02.372318 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:02.371974 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:03.370839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:03.370808 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:03.370839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:03.370835 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:03.371071 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:03.370939 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:03.371141 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:03.371077 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:04.370616 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.370443 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:04.371224 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:04.370693 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:04.504278 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.504244 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhg89" event={"ID":"8faa5588-36ea-40c7-b1b3-09cb04b05d0a","Type":"ContainerStarted","Data":"e53cf21e84344db4a250350ec7edacdb3733bbb9a47d9f49df6f45f82aecbd02"} Aug 04 00:03:04.505595 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.505572 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" event={"ID":"b91cd527bb73e98102acd9e9c144cdda","Type":"ContainerStarted","Data":"a4af95851bdc0f15d1e6d1cc2109ecd7f993e880def177a430c928f562b974cf"} Aug 04 00:03:04.507710 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.507683 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6t2wv" event={"ID":"051a04fe-978e-48ea-878f-71bf7c22419e","Type":"ContainerStarted","Data":"a5dae609b33677380370c3d5ea7b14431008b5211e9f3d83292699b69f94c1a6"} Aug 04 00:03:04.508974 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.508955 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" event={"ID":"4f4cf4fc-667a-46c8-8828-a509aafaa4ae","Type":"ContainerStarted","Data":"4102cb814f37482c8dc0be0328f536691e1efcca30b7a5c29e4ef67cf06b0fc4"} Aug 04 00:03:04.510192 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.510169 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s66z8" event={"ID":"116551c0-6d82-4e9e-be71-7a11d2ecf733","Type":"ContainerStarted","Data":"2b4b8c2fd4e1bd625dc15c5e81933c805c46dcec49b79b791f250f37226750cd"} Aug 04 00:03:04.511392 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.511370 2571 generic.go:356] "Generic (PLEG): container finished" podID="e000d8a9-3ea5-4106-96cc-2a8b36c1a680" containerID="11daf7f3a09a14d1925360db6b353655153c918a4cc9b98ecec9090dd830f051" exitCode=0 Aug 04 00:03:04.511474 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.511444 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerDied","Data":"11daf7f3a09a14d1925360db6b353655153c918a4cc9b98ecec9090dd830f051"} Aug 04 00:03:04.512855 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.512819 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tpvjb" event={"ID":"f95614fb-0969-4b83-9db8-7bbc91885d23","Type":"ContainerStarted","Data":"fbdf729b655fe0f7e4a29a86c70de33e3326dcece935613c577579a91654834f"} Aug 04 00:03:04.515867 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.515843 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"2861114f6ec95cad506f70311f362d7163f681a502a1ceb01b41fb6f2812009f"} Aug 04 00:03:04.515963 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.515906 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"f8af7398a42bc2e8aec2631726fac1f9cabd0b1fb40152d13c4fb5348e9abe96"} Aug 04 00:03:04.515963 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.515947 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"3a4aa12fe9b2d17ee0041b618f4ebded15d64053fdf9692c70f053a2c8466e34"} Aug 04 00:03:04.516070 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.515997 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"01b51fd11530ba76bbe853d7512f69432fde0154b66e5cbfad368d01db4a11a2"} Aug 04 00:03:04.517533 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.517511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kx9jg" event={"ID":"539e58ad-eeb7-4a35-8d3f-af0cebbbe186","Type":"ContainerStarted","Data":"4af04f75f03d8cca20a0df6dfb212cf38ad4894b25bb57df57832e276b91c6ee"} Aug 04 00:03:04.519711 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.519663 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xhg89" podStartSLOduration=3.120793857 podStartE2EDuration="22.519651916s" totalImagesPullingTime="19.398858059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:04.51958068 +0000 UTC m=+22.690613987" watchObservedRunningTime="2026-08-04 00:03:04.519651916 +0000 UTC m=+22.690685222" Aug 04 00:03:04.520006 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.519984 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-15.ec2.internal" podStartSLOduration=21.519976208 podStartE2EDuration="21.519976208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:02:46.482474321 +0000 UTC m=+4.653507633" watchObservedRunningTime="2026-08-04 00:03:04.519976208 +0000 UTC m=+22.691009514" Aug 04 00:03:04.532165 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.532129 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-15.ec2.internal" podStartSLOduration=21.532116157 podStartE2EDuration="21.532116157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:04.531959873 +0000 UTC m=+22.702993179" watchObservedRunningTime="2026-08-04 00:03:04.532116157 +0000 UTC m=+22.703149458" Aug 04 00:03:04.546706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.546657 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kx9jg" podStartSLOduration=3.153124826 podStartE2EDuration="22.546638809s" totalImagesPullingTime="19.393513983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:04.546224514 +0000 UTC m=+22.717257821" watchObservedRunningTime="2026-08-04 00:03:04.546638809 +0000 UTC m=+22.717672117" Aug 04 00:03:04.575603 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.575552 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6t2wv" podStartSLOduration=3.178889327 podStartE2EDuration="22.575536861s" totalImagesPullingTime="19.396647534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:04.562346865 +0000 UTC m=+22.733380173" watchObservedRunningTime="2026-08-04 00:03:04.575536861 +0000 UTC m=+22.746570167" Aug 04 00:03:04.575962 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.575939 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tpvjb" podStartSLOduration=3.192498561 podStartE2EDuration="22.575933601s" totalImagesPullingTime="19.38343504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:04.575364675 +0000 UTC m=+22.746397982" watchObservedRunningTime="2026-08-04 00:03:04.575933601 +0000 UTC m=+22.746966921" Aug 04 00:03:04.612894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:04.612846 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s66z8" podStartSLOduration=3.230043033 podStartE2EDuration="22.61283077s" totalImagesPullingTime="19.382787737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:04.612637404 +0000 UTC m=+22.783670712" watchObservedRunningTime="2026-08-04 00:03:04.61283077 +0000 UTC m=+22.783864076" Aug 04 00:03:05.370900 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.370877 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:05.371245 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.370901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:05.371245 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:05.370981 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:05.371245 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:05.371153 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:05.406673 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.406650 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 00:03:05.522488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.522401 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"bdba9b71173408beb3ab124755e7a11880ef7ef9e8b15215b07e907d302262d1"} Aug 04 00:03:05.522652 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.522516 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"f6a87082b92cbd162f8938611f2c693c20f531257f517c9c637413e47debac4d"} Aug 04 00:03:05.523713 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.523688 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zvx57" event={"ID":"c029686e-4ca6-4da0-8682-98a0bc020e86","Type":"ContainerStarted","Data":"f295610768cf89f7506c392ad5cface41cc64c8060c29132a1d425ec78a2e54c"} Aug 04 00:03:05.525538 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:05.525509 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" event={"ID":"4f4cf4fc-667a-46c8-8828-a509aafaa4ae","Type":"ContainerStarted","Data":"9d5d9544dc57249bf430b755e6fe8918471995b83710260b1bcd4dd6a1bc95a7"} Aug 04 00:03:06.300919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:06.300818 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T00:03:05.406667579Z","UUID":"9a6d9e01-7f07-48ba-b1f3-7457888d141a","Handler":null,"Name":"","Endpoint":""} Aug 04 00:03:06.302317 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:06.302297 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 00:03:06.302416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:06.302333 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 00:03:06.371353 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:06.371321 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:06.371719 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:06.371420 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:07.370845 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:07.370598 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:07.371142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:07.370630 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:07.371142 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:07.370970 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:07.371142 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:07.371038 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:07.531425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:07.531391 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" event={"ID":"4f4cf4fc-667a-46c8-8828-a509aafaa4ae","Type":"ContainerStarted","Data":"c2247a4f37ee644fb5f573e289ddebeb7fd71577b23be1168b539e7a7ea61767"} Aug 04 00:03:07.535210 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:07.535176 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"36ea6626ba9160d1b1e7a1f038763a761e8fcf9c0ff4611710de49b732bbb319"} Aug 04 00:03:07.552112 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:07.552048 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjk4b" podStartSLOduration=3.845847169 podStartE2EDuration="25.552033806s" totalImagesPullingTime="21.706186637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:07.552007725 +0000 UTC m=+25.723041031" watchObservedRunningTime="2026-08-04 00:03:07.552033806 +0000 UTC m=+25.723067112" Aug 04 00:03:07.552368 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:07.552347 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zvx57" podStartSLOduration=6.198883772 podStartE2EDuration="25.552341616s" totalImagesPullingTime="19.353457844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:05.536839836 +0000 UTC m=+23.707873142" watchObservedRunningTime="2026-08-04 00:03:07.552341616 +0000 UTC m=+25.723374922" Aug 04 00:03:08.124000 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:08.123960 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:03:08.124727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:08.124711 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:03:08.370771 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:08.370724 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:08.370930 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:08.370856 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:08.537918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:08.537827 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:03:08.538589 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:08.538312 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s66z8" Aug 04 00:03:09.371542 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.371312 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:09.371763 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.371350 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:09.371763 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:09.371644 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:09.371763 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:09.371690 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:09.543954 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.543916 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" event={"ID":"7357fd0a-8ae8-4dca-9466-78f85c4efa53","Type":"ContainerStarted","Data":"cf019b65b829f8b4cc641d8f7d4b21f1bae87a9cdb98d2921e80869715f03606"} Aug 04 00:03:09.544696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.544231 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:03:09.544696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.544335 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:03:09.544696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.544456 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:03:09.562385 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.562356 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:03:09.563114 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.563076 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:03:09.595595 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:09.595547 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" podStartSLOduration=8.161290721 podStartE2EDuration="27.595525539s" totalImagesPullingTime="19.434234818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:09.571726501 +0000 UTC m=+27.742759809" watchObservedRunningTime="2026-08-04 00:03:09.595525539 +0000 UTC m=+27.766558848" Aug 04 00:03:10.370499 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.370407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:10.370639 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:10.370525 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:10.966182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.966158 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9wzbx"] Aug 04 00:03:10.966569 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.966302 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:10.966569 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:10.966423 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:10.968658 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.968628 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sbgfn"] Aug 04 00:03:10.968818 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.968730 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:10.968890 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:10.968836 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:10.969438 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.969414 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tbpf"] Aug 04 00:03:10.969543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:10.969510 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:10.969615 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:10.969592 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:11.550052 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:11.549798 2571 generic.go:356] "Generic (PLEG): container finished" podID="e000d8a9-3ea5-4106-96cc-2a8b36c1a680" containerID="87e89fcf55d538ccc2c95d4284698dcdd29d6023f4356aedf3c83fe8b89b8f53" exitCode=0 Aug 04 00:03:11.550245 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:11.549884 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerDied","Data":"87e89fcf55d538ccc2c95d4284698dcdd29d6023f4356aedf3c83fe8b89b8f53"} Aug 04 00:03:12.371272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:12.371232 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:12.371844 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:12.371331 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:12.371844 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:12.371399 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:12.371844 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:12.371485 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:12.371844 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:12.371568 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:12.371844 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:12.371665 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:14.370903 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:14.370873 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:14.371500 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:14.370986 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:14.371500 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:14.370873 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:14.371500 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:14.371021 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:14.371500 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:14.371121 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:14.371500 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:14.371195 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:14.559708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:14.559622 2571 generic.go:356] "Generic (PLEG): container finished" podID="e000d8a9-3ea5-4106-96cc-2a8b36c1a680" containerID="5f3f5beb75cbee542b1b0c2cbcb79837af1863321edb65cfe5c469ead04aaa7d" exitCode=0 Aug 04 00:03:14.559708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:14.559676 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerDied","Data":"5f3f5beb75cbee542b1b0c2cbcb79837af1863321edb65cfe5c469ead04aaa7d"} Aug 04 00:03:15.564306 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:15.564274 2571 generic.go:356] "Generic (PLEG): container finished" podID="e000d8a9-3ea5-4106-96cc-2a8b36c1a680" containerID="c2ad304f40f960d5ab61c2aa064b7abebe0727390a44e1442e2851aba0fce79b" exitCode=0 Aug 04 00:03:15.564306 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:15.564309 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerDied","Data":"c2ad304f40f960d5ab61c2aa064b7abebe0727390a44e1442e2851aba0fce79b"} Aug 04 00:03:15.977708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:15.977607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:15.977708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:15.977658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:15.977938 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:15.977754 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:15.977938 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:15.977755 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 00:03:15.977938 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:15.977815 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:47.977798087 +0000 UTC m=+66.148831385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 00:03:15.977938 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:15.977832 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret podName:8ae9a103-10de-4451-bb24-97976795b3c5 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:47.977823864 +0000 UTC m=+66.148857152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret") pod "global-pull-secret-syncer-sbgfn" (UID: "8ae9a103-10de-4451-bb24-97976795b3c5") : object "kube-system"/"original-pull-secret" not registered Aug 04 00:03:16.078254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.078214 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:16.078425 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.078399 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 00:03:16.078425 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.078425 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 00:03:16.078500 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.078435 2571 projected.go:196] Error preparing data for projected volume kube-api-access-kdlwb for pod openshift-network-diagnostics/network-check-target-2tbpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:16.078500 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.078494 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb podName:5b785f8d-c849-4bcc-8f5d-d1f073b33ce1 nodeName:}" failed. No retries permitted until 2026-08-04 00:03:48.078479284 +0000 UTC m=+66.249512582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdlwb" (UniqueName: "kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb") pod "network-check-target-2tbpf" (UID: "5b785f8d-c849-4bcc-8f5d-d1f073b33ce1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 00:03:16.371257 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.371213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:16.371257 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.371236 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:16.371257 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.371238 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:16.371480 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.371340 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tbpf" podUID="5b785f8d-c849-4bcc-8f5d-d1f073b33ce1" Aug 04 00:03:16.371480 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.371394 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sbgfn" podUID="8ae9a103-10de-4451-bb24-97976795b3c5" Aug 04 00:03:16.371480 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.371459 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:03:16.611329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.611296 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-15.ec2.internal" event="NodeReady" Aug 04 00:03:16.611689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.611416 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 00:03:16.652642 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.652567 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-htm5p"] Aug 04 00:03:16.656595 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.656570 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lb2n9"] Aug 04 00:03:16.656702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.656683 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.658885 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.658864 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 00:03:16.659014 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.658870 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7ds4t\"" Aug 04 00:03:16.659187 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.659170 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 00:03:16.660708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.660691 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.662691 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.662675 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 00:03:16.662790 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.662773 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 00:03:16.662847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.662788 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 00:03:16.662908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.662853 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4xmt\"" Aug 04 00:03:16.664832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.664810 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-htm5p"] Aug 04 00:03:16.668646 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.668596 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lb2n9"] Aug 04 00:03:16.783247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.783211 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277504f0-4353-4cca-9cc2-3610dbdd9fdc-tmp-dir\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.783247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.783245 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b09fb90-1aef-4889-8783-bd14933ff063-cert\") pod \"ingress-canary-lb2n9\" (UID: \"9b09fb90-1aef-4889-8783-bd14933ff063\") " pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.783463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.783279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvst\" (UniqueName: \"kubernetes.io/projected/9b09fb90-1aef-4889-8783-bd14933ff063-kube-api-access-9rvst\") pod \"ingress-canary-lb2n9\" (UID: \"9b09fb90-1aef-4889-8783-bd14933ff063\") " pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.783463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.783303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.783463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.783330 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/277504f0-4353-4cca-9cc2-3610dbdd9fdc-config-volume\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.783463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.783352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4p8k\" (UniqueName: \"kubernetes.io/projected/277504f0-4353-4cca-9cc2-3610dbdd9fdc-kube-api-access-l4p8k\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.884361 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884328 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/277504f0-4353-4cca-9cc2-3610dbdd9fdc-config-volume\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.884361 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4p8k\" (UniqueName: \"kubernetes.io/projected/277504f0-4353-4cca-9cc2-3610dbdd9fdc-kube-api-access-l4p8k\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.884576 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884421 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277504f0-4353-4cca-9cc2-3610dbdd9fdc-tmp-dir\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.884576 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b09fb90-1aef-4889-8783-bd14933ff063-cert\") pod \"ingress-canary-lb2n9\" (UID: \"9b09fb90-1aef-4889-8783-bd14933ff063\") " pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.884576 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvst\" (UniqueName: \"kubernetes.io/projected/9b09fb90-1aef-4889-8783-bd14933ff063-kube-api-access-9rvst\") pod \"ingress-canary-lb2n9\" (UID: \"9b09fb90-1aef-4889-8783-bd14933ff063\") " pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.884576 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884471 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.884576 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.884571 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:16.884806 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:16.884636 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:17.384618183 +0000 UTC m=+35.555651482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:16.884894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884825 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277504f0-4353-4cca-9cc2-3610dbdd9fdc-tmp-dir\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.884966 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.884948 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/277504f0-4353-4cca-9cc2-3610dbdd9fdc-config-volume\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.888701 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.888576 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b09fb90-1aef-4889-8783-bd14933ff063-cert\") pod \"ingress-canary-lb2n9\" (UID: \"9b09fb90-1aef-4889-8783-bd14933ff063\") " pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.892858 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.892828 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvst\" (UniqueName: \"kubernetes.io/projected/9b09fb90-1aef-4889-8783-bd14933ff063-kube-api-access-9rvst\") pod \"ingress-canary-lb2n9\" (UID: \"9b09fb90-1aef-4889-8783-bd14933ff063\") " pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:16.892945 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.892924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4p8k\" (UniqueName: \"kubernetes.io/projected/277504f0-4353-4cca-9cc2-3610dbdd9fdc-kube-api-access-l4p8k\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:16.973222 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:16.973126 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lb2n9" Aug 04 00:03:17.106742 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:17.106714 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lb2n9"] Aug 04 00:03:17.112970 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:03:17.112941 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b09fb90_1aef_4889_8783_bd14933ff063.slice/crio-30ff68143e4606b025edb325e8fbbca82be6f336b6f5fe28a86725ca664841c1 WatchSource:0}: Error finding container 30ff68143e4606b025edb325e8fbbca82be6f336b6f5fe28a86725ca664841c1: Status 404 returned error can't find the container with id 30ff68143e4606b025edb325e8fbbca82be6f336b6f5fe28a86725ca664841c1 Aug 04 00:03:17.389410 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:17.389379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:17.389625 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:17.389491 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:17.389625 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:17.389546 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:18.389532698 +0000 UTC m=+36.560565983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:17.570429 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:17.570391 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lb2n9" event={"ID":"9b09fb90-1aef-4889-8783-bd14933ff063","Type":"ContainerStarted","Data":"30ff68143e4606b025edb325e8fbbca82be6f336b6f5fe28a86725ca664841c1"} Aug 04 00:03:18.374768 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.374736 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:18.375351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.375144 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:18.375451 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.375412 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:18.378615 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.378500 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjt6l\"" Aug 04 00:03:18.378615 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.378548 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 00:03:18.378615 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.378597 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 00:03:18.378615 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.378553 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 00:03:18.378925 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.378602 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9tct\"" Aug 04 00:03:18.378925 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.378548 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 00:03:18.399577 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:18.399555 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:18.399714 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:18.399699 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:18.399770 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:18.399751 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:20.399737892 +0000 UTC m=+38.570771180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:20.414021 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:20.413799 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:20.414461 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:20.413956 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:20.414461 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:20.414127 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:24.414089226 +0000 UTC m=+42.585122528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:20.579023 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:20.578984 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lb2n9" event={"ID":"9b09fb90-1aef-4889-8783-bd14933ff063","Type":"ContainerStarted","Data":"9ecd3ed3867e3e74f7e3ec57a11bd151cd22d907c3da42c010b7d2a8bf88c073"} Aug 04 00:03:20.595792 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:20.595739 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lb2n9" podStartSLOduration=1.966024056 podStartE2EDuration="4.595723051s" totalImagesPullingTime="2.629698995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:20.595369465 +0000 UTC m=+38.766402773" watchObservedRunningTime="2026-08-04 00:03:20.595723051 +0000 UTC m=+38.766756359" Aug 04 00:03:22.585982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:22.585950 2571 generic.go:356] "Generic (PLEG): container finished" podID="e000d8a9-3ea5-4106-96cc-2a8b36c1a680" containerID="93eb85406eb61483c82496bcf72201c2036e6cb7aee5629fa93dd3889a6001bb" exitCode=0 Aug 04 00:03:22.586747 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:22.586001 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerDied","Data":"93eb85406eb61483c82496bcf72201c2036e6cb7aee5629fa93dd3889a6001bb"} Aug 04 00:03:23.590996 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:23.590958 2571 generic.go:356] "Generic (PLEG): container finished" podID="e000d8a9-3ea5-4106-96cc-2a8b36c1a680" containerID="e526faf3677f496bfbabb05294eaa4de06450cffe319cf6da62cdd201d459b2e" exitCode=0 Aug 04 00:03:23.591394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:23.591029 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerDied","Data":"e526faf3677f496bfbabb05294eaa4de06450cffe319cf6da62cdd201d459b2e"} Aug 04 00:03:24.441913 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:24.441873 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:24.442117 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:24.442001 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:24.442117 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:24.442055 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:32.44204046 +0000 UTC m=+50.613073748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:24.596932 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:24.596893 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46gf6" event={"ID":"e000d8a9-3ea5-4106-96cc-2a8b36c1a680","Type":"ContainerStarted","Data":"bc4cf9a9ab518628e2ccd339400b291bb407452207d1d85c422e34c811b1183e"} Aug 04 00:03:24.620901 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:24.620850 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-46gf6" podStartSLOduration=7.218079565 podStartE2EDuration="42.620830335s" totalImagesPullingTime="35.40275077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:24.620196379 +0000 UTC m=+42.791229686" watchObservedRunningTime="2026-08-04 00:03:24.620830335 +0000 UTC m=+42.791863644" Aug 04 00:03:32.499749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:32.499704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:32.500180 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:32.499823 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:32.500180 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:32.499885 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:03:48.499871653 +0000 UTC m=+66.670904938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:41.559961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:41.559933 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v42gn" Aug 04 00:03:48.000400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.000355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:48.000918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.000414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:03:48.002718 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.002693 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 00:03:48.002850 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.002831 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 00:03:48.011688 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:48.011670 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:03:48.011761 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:48.011727 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:04:52.011711481 +0000 UTC m=+130.182744767 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : secret "metrics-daemon-secret" not found Aug 04 00:03:48.015258 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.015235 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ae9a103-10de-4451-bb24-97976795b3c5-original-pull-secret\") pod \"global-pull-secret-syncer-sbgfn\" (UID: \"8ae9a103-10de-4451-bb24-97976795b3c5\") " pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:48.090057 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.090026 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sbgfn" Aug 04 00:03:48.101412 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.101390 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:48.104561 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.104538 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 00:03:48.113900 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.113876 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 00:03:48.123979 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.123960 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlwb\" (UniqueName: \"kubernetes.io/projected/5b785f8d-c849-4bcc-8f5d-d1f073b33ce1-kube-api-access-kdlwb\") pod \"network-check-target-2tbpf\" (UID: \"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1\") " pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:48.213271 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.213235 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sbgfn"] Aug 04 00:03:48.217172 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:03:48.217138 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae9a103_10de_4451_bb24_97976795b3c5.slice/crio-0dfdbd7659fa2cbf652b155bd836dc7060b3bf07ef1e3bb02df66225af768595 WatchSource:0}: Error finding container 0dfdbd7659fa2cbf652b155bd836dc7060b3bf07ef1e3bb02df66225af768595: Status 404 returned error can't find the container with id 0dfdbd7659fa2cbf652b155bd836dc7060b3bf07ef1e3bb02df66225af768595 Aug 04 00:03:48.401850 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.401818 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qjt6l\"" Aug 04 00:03:48.410160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.410136 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:48.503755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.503725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:03:48.503930 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:48.503908 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:03:48.504017 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:03:48.504004 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:04:20.503976963 +0000 UTC m=+98.675010271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:03:48.541050 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.541021 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tbpf"] Aug 04 00:03:48.544356 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:03:48.544327 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b785f8d_c849_4bcc_8f5d_d1f073b33ce1.slice/crio-7a7c1df8a1e5796f9eb9cffb16c948d42cd8dd66753692c5de58955495095ca6 WatchSource:0}: Error finding container 7a7c1df8a1e5796f9eb9cffb16c948d42cd8dd66753692c5de58955495095ca6: Status 404 returned error can't find the container with id 7a7c1df8a1e5796f9eb9cffb16c948d42cd8dd66753692c5de58955495095ca6 Aug 04 00:03:48.653586 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.653499 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tbpf" event={"ID":"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1","Type":"ContainerStarted","Data":"7a7c1df8a1e5796f9eb9cffb16c948d42cd8dd66753692c5de58955495095ca6"} Aug 04 00:03:48.654800 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:48.654773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sbgfn" event={"ID":"8ae9a103-10de-4451-bb24-97976795b3c5","Type":"ContainerStarted","Data":"0dfdbd7659fa2cbf652b155bd836dc7060b3bf07ef1e3bb02df66225af768595"} Aug 04 00:03:53.670833 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:53.670795 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tbpf" event={"ID":"5b785f8d-c849-4bcc-8f5d-d1f073b33ce1","Type":"ContainerStarted","Data":"377aa9fee2384cd02f815a2f1e8d7512e19d5a88ee5eeac3e7a437355c2fa664"} Aug 04 00:03:53.671365 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:53.670927 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:03:53.672379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:53.672357 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sbgfn" event={"ID":"8ae9a103-10de-4451-bb24-97976795b3c5","Type":"ContainerStarted","Data":"8e7590baf03a5bc12ad98c2e97fb383652f64503768bb4e3dde3ee4327280f13"} Aug 04 00:03:53.685654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:53.685612 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2tbpf" podStartSLOduration=67.458564093 podStartE2EDuration="1m11.685600495s" totalImagesPullingTime="4.227036402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:53.685216457 +0000 UTC m=+71.856249775" watchObservedRunningTime="2026-08-04 00:03:53.685600495 +0000 UTC m=+71.856633802" Aug 04 00:03:53.698795 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:03:53.698757 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sbgfn" podStartSLOduration=67.139122079 podStartE2EDuration="1m11.698746472s" totalImagesPullingTime="4.559624393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:03:53.698219022 +0000 UTC m=+71.869252342" watchObservedRunningTime="2026-08-04 00:03:53.698746472 +0000 UTC m=+71.869779779" Aug 04 00:04:03.497663 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.497631 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk"] Aug 04 00:04:03.499598 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.499583 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.502715 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.502693 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 00:04:03.502715 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.502706 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 00:04:03.502904 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.502699 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 00:04:03.502904 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.502693 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 04 00:04:03.502904 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.502828 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hjm22\"" Aug 04 00:04:03.509964 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.509942 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk"] Aug 04 00:04:03.601387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.601343 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc495\" (UniqueName: \"kubernetes.io/projected/c788365d-e2d4-43d6-be4e-bc4bc7a28f5a-kube-api-access-vc495\") pod \"managed-serviceaccount-addon-agent-7dd58cd484-krpkk\" (UID: \"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.601387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.601388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c788365d-e2d4-43d6-be4e-bc4bc7a28f5a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd58cd484-krpkk\" (UID: \"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.702042 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.702008 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c788365d-e2d4-43d6-be4e-bc4bc7a28f5a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd58cd484-krpkk\" (UID: \"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.702240 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.702095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc495\" (UniqueName: \"kubernetes.io/projected/c788365d-e2d4-43d6-be4e-bc4bc7a28f5a-kube-api-access-vc495\") pod \"managed-serviceaccount-addon-agent-7dd58cd484-krpkk\" (UID: \"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.704506 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.704476 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c788365d-e2d4-43d6-be4e-bc4bc7a28f5a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd58cd484-krpkk\" (UID: \"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.709803 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.709778 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc495\" (UniqueName: \"kubernetes.io/projected/c788365d-e2d4-43d6-be4e-bc4bc7a28f5a-kube-api-access-vc495\") pod \"managed-serviceaccount-addon-agent-7dd58cd484-krpkk\" (UID: \"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.819887 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.819853 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" Aug 04 00:04:03.937134 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:03.937081 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk"] Aug 04 00:04:03.941265 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:04:03.941228 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc788365d_e2d4_43d6_be4e_bc4bc7a28f5a.slice/crio-8514ef18876bb6b668dd7a71367bfce254a32c776eadd32ee90cffcad4757cee WatchSource:0}: Error finding container 8514ef18876bb6b668dd7a71367bfce254a32c776eadd32ee90cffcad4757cee: Status 404 returned error can't find the container with id 8514ef18876bb6b668dd7a71367bfce254a32c776eadd32ee90cffcad4757cee Aug 04 00:04:04.700247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:04.700208 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" event={"ID":"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a","Type":"ContainerStarted","Data":"8514ef18876bb6b668dd7a71367bfce254a32c776eadd32ee90cffcad4757cee"} Aug 04 00:04:07.708812 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:07.708777 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" event={"ID":"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a","Type":"ContainerStarted","Data":"73ce8d6b5ea785324a524959589933500d67e31e963aec4620d193918924fd46"} Aug 04 00:04:07.726803 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:07.724681 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" podStartSLOduration=1.5229357449999998 podStartE2EDuration="4.724662692s" totalImagesPullingTime="3.201726947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:04:07.723999739 +0000 UTC m=+85.895033049" watchObservedRunningTime="2026-08-04 00:04:07.724662692 +0000 UTC m=+85.895696000" Aug 04 00:04:20.517425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:20.517370 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:04:20.517952 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:04:20.517536 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:04:20.517952 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:04:20.517604 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:05:24.517587499 +0000 UTC m=+162.688620784 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:04:24.676997 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:24.676964 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2tbpf" Aug 04 00:04:52.032031 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:04:52.031974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:04:52.032649 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:04:52.032170 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 00:04:52.032649 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:04:52.032266 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs podName:3ba07224-77d2-4138-873b-4278e1b0cbb5 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.032245226 +0000 UTC m=+252.203278525 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs") pod "network-metrics-daemon-9wzbx" (UID: "3ba07224-77d2-4138-873b-4278e1b0cbb5") : secret "metrics-daemon-secret" not found Aug 04 00:05:19.666591 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:05:19.666538 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-htm5p" podUID="277504f0-4353-4cca-9cc2-3610dbdd9fdc" Aug 04 00:05:19.878512 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:05:19.878481 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htm5p" Aug 04 00:05:21.405320 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:05:21.405264 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9wzbx" podUID="3ba07224-77d2-4138-873b-4278e1b0cbb5" Aug 04 00:05:24.546121 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:05:24.546050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:05:24.546602 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:05:24.546231 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 00:05:24.546602 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:05:24.546320 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls podName:277504f0-4353-4cca-9cc2-3610dbdd9fdc nodeName:}" failed. No retries permitted until 2026-08-04 00:07:26.546298189 +0000 UTC m=+284.717331478 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls") pod "dns-default-htm5p" (UID: "277504f0-4353-4cca-9cc2-3610dbdd9fdc") : secret "dns-default-metrics-tls" not found Aug 04 00:05:33.371359 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:05:33.371324 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:06:07.996656 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:07.996617 2571 generic.go:356] "Generic (PLEG): container finished" podID="c788365d-e2d4-43d6-be4e-bc4bc7a28f5a" containerID="73ce8d6b5ea785324a524959589933500d67e31e963aec4620d193918924fd46" exitCode=255 Aug 04 00:06:07.997076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:07.996692 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" event={"ID":"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a","Type":"ContainerDied","Data":"73ce8d6b5ea785324a524959589933500d67e31e963aec4620d193918924fd46"} Aug 04 00:06:08.002060 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:08.002036 2571 scope.go:117] "RemoveContainer" containerID="73ce8d6b5ea785324a524959589933500d67e31e963aec4620d193918924fd46" Aug 04 00:06:09.000973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:09.000938 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd58cd484-krpkk" event={"ID":"c788365d-e2d4-43d6-be4e-bc4bc7a28f5a","Type":"ContainerStarted","Data":"5f6eaf3465c25ce02b97921796b860a63d6beea37688c618d3b026272f9848ef"} Aug 04 00:06:34.283275 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.283236 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79d7f977cd-44xk6"] Aug 04 00:06:34.285749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.285726 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8"] Aug 04 00:06:34.285881 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.285864 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.287981 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.287943 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56c65ccbd6-nsg9s"] Aug 04 00:06:34.288079 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.288065 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.288538 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.288519 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbxcz\"" Aug 04 00:06:34.288618 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.288527 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 00:06:34.288922 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.288902 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 00:06:34.288922 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.288914 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 00:06:34.290613 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.290599 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.291382 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.291335 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.291736 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.291613 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.291736 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.291636 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 00:06:34.291967 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.291948 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wn4l8\"" Aug 04 00:06:34.292278 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.292259 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 00:06:34.292617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.292605 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 00:06:34.293158 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.292825 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 00:06:34.293540 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.293521 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jh8rb\"" Aug 04 00:06:34.293625 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.293590 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 00:06:34.293890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.293874 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 00:06:34.294142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.294129 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 00:06:34.294405 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.294388 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 00:06:34.297946 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.297925 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 00:06:34.303751 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.303478 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8"] Aug 04 00:06:34.304618 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.304595 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d7f977cd-44xk6"] Aug 04 00:06:34.310272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.310249 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56c65ccbd6-nsg9s"] Aug 04 00:06:34.397559 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397527 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2be9d4e-c830-49a6-91fe-b1d98984dd98-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.397559 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397561 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-trusted-ca\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-stats-auth\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397599 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-image-registry-private-configuration\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397616 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9l9\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-kube-api-access-xw9l9\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-certificates\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397684 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cn88\" (UniqueName: \"kubernetes.io/projected/a2be9d4e-c830-49a6-91fe-b1d98984dd98-kube-api-access-6cn88\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-default-certificate\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a88aaff-5987-46b0-9fce-c358a2e75f6f-ca-trust-extracted\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.397784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397765 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2be9d4e-c830-49a6-91fe-b1d98984dd98-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.398195 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.398195 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-bound-sa-token\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.398195 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397933 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-installation-pull-secrets\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.398195 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397948 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.398195 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.397984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx954\" (UniqueName: \"kubernetes.io/projected/272f48c6-278a-409c-a1e5-a300dcb4a2aa-kube-api-access-zx954\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.469083 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.469047 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-whqmh"] Aug 04 00:06:34.471724 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.471709 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:34.474275 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.474252 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 04 00:06:34.474396 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.474276 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 04 00:06:34.474396 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.474291 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dxw7k\"" Aug 04 00:06:34.482769 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.482746 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-whqmh"] Aug 04 00:06:34.499141 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499110 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.499272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499156 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-bound-sa-token\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.499272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499198 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-installation-pull-secrets\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.499272 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.499211 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:06:34.499272 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.499258 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d7f977cd-44xk6: secret "image-registry-tls" not found Aug 04 00:06:34.499488 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.499319 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls podName:5a88aaff-5987-46b0-9fce-c358a2e75f6f nodeName:}" failed. No retries permitted until 2026-08-04 00:06:34.999303548 +0000 UTC m=+233.170336832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls") pod "image-registry-79d7f977cd-44xk6" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f") : secret "image-registry-tls" not found Aug 04 00:06:34.499488 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.499346 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:34.999328989 +0000 UTC m=+233.170362290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : configmap references non-existent config key: service-ca.crt Aug 04 00:06:34.499488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.499488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499410 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx954\" (UniqueName: \"kubernetes.io/projected/272f48c6-278a-409c-a1e5-a300dcb4a2aa-kube-api-access-zx954\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.499488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2be9d4e-c830-49a6-91fe-b1d98984dd98-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.499488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499475 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-trusted-ca\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.499808 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-stats-auth\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.499808 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-image-registry-private-configuration\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.499808 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499555 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9l9\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-kube-api-access-xw9l9\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.499808 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-certificates\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.500022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.499967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cn88\" (UniqueName: \"kubernetes.io/projected/a2be9d4e-c830-49a6-91fe-b1d98984dd98-kube-api-access-6cn88\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.500022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-default-certificate\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.500159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500035 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.500159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500050 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2be9d4e-c830-49a6-91fe-b1d98984dd98-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.500159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a88aaff-5987-46b0-9fce-c358a2e75f6f-ca-trust-extracted\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.500159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2be9d4e-c830-49a6-91fe-b1d98984dd98-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.500437 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500418 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-certificates\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.500568 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500452 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-trusted-ca\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.500657 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.500508 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 00:06:34.500657 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.500653 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:35.000636471 +0000 UTC m=+233.171669771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : secret "router-metrics-certs-default" not found Aug 04 00:06:34.500771 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.500710 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a88aaff-5987-46b0-9fce-c358a2e75f6f-ca-trust-extracted\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.502232 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.502212 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-installation-pull-secrets\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.502328 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.502312 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2be9d4e-c830-49a6-91fe-b1d98984dd98-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.502428 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.502410 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-image-registry-private-configuration\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.502524 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.502504 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-stats-auth\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.502650 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.502632 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-default-certificate\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.509652 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.509628 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9l9\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-kube-api-access-xw9l9\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.509809 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.509779 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx954\" (UniqueName: \"kubernetes.io/projected/272f48c6-278a-409c-a1e5-a300dcb4a2aa-kube-api-access-zx954\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:34.510164 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.510146 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-bound-sa-token\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:34.510866 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.510848 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cn88\" (UniqueName: \"kubernetes.io/projected/a2be9d4e-c830-49a6-91fe-b1d98984dd98-kube-api-access-6cn88\") pod \"kube-storage-version-migrator-operator-7878f4b88c-z2hl8\" (UID: \"a2be9d4e-c830-49a6-91fe-b1d98984dd98\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.600521 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.600452 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:34.600656 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.600538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/788bda68-1682-402e-8cea-39859d7323e3-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:34.603240 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.603221 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" Aug 04 00:06:34.701114 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.701075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/788bda68-1682-402e-8cea-39859d7323e3-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:34.701251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.701148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:34.701302 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.701264 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 00:06:34.701347 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:34.701314 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert podName:788bda68-1682-402e-8cea-39859d7323e3 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:35.201301128 +0000 UTC m=+233.372334413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-whqmh" (UID: "788bda68-1682-402e-8cea-39859d7323e3") : secret "networking-console-plugin-cert" not found Aug 04 00:06:34.701809 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.701787 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/788bda68-1682-402e-8cea-39859d7323e3-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:34.719511 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:34.719485 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8"] Aug 04 00:06:34.722519 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:06:34.722492 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2be9d4e_c830_49a6_91fe_b1d98984dd98.slice/crio-10d98fbe8ed8bf79da4c4d024ee6eff58a0bd2f4a394674e438d11c66b270686 WatchSource:0}: Error finding container 10d98fbe8ed8bf79da4c4d024ee6eff58a0bd2f4a394674e438d11c66b270686: Status 404 returned error can't find the container with id 10d98fbe8ed8bf79da4c4d024ee6eff58a0bd2f4a394674e438d11c66b270686 Aug 04 00:06:35.004096 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:35.004009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:35.004096 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:35.004054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:35.004096 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:35.004096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:35.004363 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.004198 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:06:35.004363 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.004215 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 00:06:35.004363 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.004220 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d7f977cd-44xk6: secret "image-registry-tls" not found Aug 04 00:06:35.004363 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.004241 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:36.004225038 +0000 UTC m=+234.175258328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : configmap references non-existent config key: service-ca.crt Aug 04 00:06:35.004363 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.004261 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:36.004254106 +0000 UTC m=+234.175287391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : secret "router-metrics-certs-default" not found Aug 04 00:06:35.004363 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.004271 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls podName:5a88aaff-5987-46b0-9fce-c358a2e75f6f nodeName:}" failed. No retries permitted until 2026-08-04 00:06:36.004266031 +0000 UTC m=+234.175299317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls") pod "image-registry-79d7f977cd-44xk6" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f") : secret "image-registry-tls" not found Aug 04 00:06:35.062543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:35.062514 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" event={"ID":"a2be9d4e-c830-49a6-91fe-b1d98984dd98","Type":"ContainerStarted","Data":"10d98fbe8ed8bf79da4c4d024ee6eff58a0bd2f4a394674e438d11c66b270686"} Aug 04 00:06:35.206021 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:35.205986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:35.206198 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.206117 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 00:06:35.206198 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:35.206170 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert podName:788bda68-1682-402e-8cea-39859d7323e3 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:36.206155547 +0000 UTC m=+234.377188832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-whqmh" (UID: "788bda68-1682-402e-8cea-39859d7323e3") : secret "networking-console-plugin-cert" not found Aug 04 00:06:36.012636 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:36.012602 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:36.012665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:36.012716 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.012774 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.012841 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:38.01282519 +0000 UTC m=+236.183858476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : configmap references non-existent config key: service-ca.crt Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.012862 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:38.012851683 +0000 UTC m=+236.183884982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : secret "router-metrics-certs-default" not found Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.012866 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.012889 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d7f977cd-44xk6: secret "image-registry-tls" not found Aug 04 00:06:36.013135 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.012943 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls podName:5a88aaff-5987-46b0-9fce-c358a2e75f6f nodeName:}" failed. No retries permitted until 2026-08-04 00:06:38.012926213 +0000 UTC m=+236.183959500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls") pod "image-registry-79d7f977cd-44xk6" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f") : secret "image-registry-tls" not found Aug 04 00:06:36.214587 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:36.214547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:36.214750 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.214721 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 00:06:36.214871 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:36.214805 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert podName:788bda68-1682-402e-8cea-39859d7323e3 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:38.214782803 +0000 UTC m=+236.385816094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-whqmh" (UID: "788bda68-1682-402e-8cea-39859d7323e3") : secret "networking-console-plugin-cert" not found Aug 04 00:06:37.069852 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.069817 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" event={"ID":"a2be9d4e-c830-49a6-91fe-b1d98984dd98","Type":"ContainerStarted","Data":"dcb4352eea7b0298504ea4d71c68f33592aa53fb30a08950fec934652c45e993"} Aug 04 00:06:37.088238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.088183 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" podStartSLOduration=1.181258275 podStartE2EDuration="3.088168712s" totalImagesPullingTime="1.906910437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:37.087302843 +0000 UTC m=+235.258336153" watchObservedRunningTime="2026-08-04 00:06:37.088168712 +0000 UTC m=+235.259202088" Aug 04 00:06:37.858721 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.858687 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm"] Aug 04 00:06:37.861694 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.861675 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" Aug 04 00:06:37.864093 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.864070 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 04 00:06:37.864218 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.864089 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 04 00:06:37.866201 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.866171 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jgrk\"" Aug 04 00:06:37.870170 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.870149 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm"] Aug 04 00:06:37.927664 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:37.927635 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgh7f\" (UniqueName: \"kubernetes.io/projected/cf63f97c-c35a-4b5d-978a-d6178e0c45f8-kube-api-access-pgh7f\") pod \"migrator-68bc745d87-qxvlm\" (UID: \"cf63f97c-c35a-4b5d-978a-d6178e0c45f8\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" Aug 04 00:06:38.028924 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.028891 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:38.029097 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.028942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:38.029097 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.028978 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgh7f\" (UniqueName: \"kubernetes.io/projected/cf63f97c-c35a-4b5d-978a-d6178e0c45f8-kube-api-access-pgh7f\") pod \"migrator-68bc745d87-qxvlm\" (UID: \"cf63f97c-c35a-4b5d-978a-d6178e0c45f8\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" Aug 04 00:06:38.029097 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.029034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:38.029097 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.029078 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 00:06:38.029294 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.029138 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.029093869 +0000 UTC m=+240.200127154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : configmap references non-existent config key: service-ca.crt Aug 04 00:06:38.029294 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.029148 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:06:38.029294 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.029163 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d7f977cd-44xk6: secret "image-registry-tls" not found Aug 04 00:06:38.029294 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.029170 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.029157226 +0000 UTC m=+240.200190511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : secret "router-metrics-certs-default" not found Aug 04 00:06:38.029294 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.029207 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls podName:5a88aaff-5987-46b0-9fce-c358a2e75f6f nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.029190224 +0000 UTC m=+240.200223512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls") pod "image-registry-79d7f977cd-44xk6" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f") : secret "image-registry-tls" not found Aug 04 00:06:38.039012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.038986 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgh7f\" (UniqueName: \"kubernetes.io/projected/cf63f97c-c35a-4b5d-978a-d6178e0c45f8-kube-api-access-pgh7f\") pod \"migrator-68bc745d87-qxvlm\" (UID: \"cf63f97c-c35a-4b5d-978a-d6178e0c45f8\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" Aug 04 00:06:38.171499 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.171418 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" Aug 04 00:06:38.231348 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.231309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:38.231522 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.231433 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 00:06:38.231522 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:38.231488 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert podName:788bda68-1682-402e-8cea-39859d7323e3 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.231473177 +0000 UTC m=+240.402506461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-whqmh" (UID: "788bda68-1682-402e-8cea-39859d7323e3") : secret "networking-console-plugin-cert" not found Aug 04 00:06:38.296022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:38.295994 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm"] Aug 04 00:06:38.298535 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:06:38.298510 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf63f97c_c35a_4b5d_978a_d6178e0c45f8.slice/crio-e9a873ee88ad723c7a4509333a2399eb41808229746154f05dc3084cf25cb236 WatchSource:0}: Error finding container e9a873ee88ad723c7a4509333a2399eb41808229746154f05dc3084cf25cb236: Status 404 returned error can't find the container with id e9a873ee88ad723c7a4509333a2399eb41808229746154f05dc3084cf25cb236 Aug 04 00:06:39.078780 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.078741 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" event={"ID":"cf63f97c-c35a-4b5d-978a-d6178e0c45f8","Type":"ContainerStarted","Data":"e9a873ee88ad723c7a4509333a2399eb41808229746154f05dc3084cf25cb236"} Aug 04 00:06:39.153932 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.153899 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zw2f8"] Aug 04 00:06:39.156740 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.156722 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.160040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.160018 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 00:06:39.160040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.160038 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 00:06:39.160224 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.160121 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbnpj\"" Aug 04 00:06:39.161304 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.161282 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 00:06:39.161401 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.161304 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 00:06:39.166701 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.166681 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zw2f8"] Aug 04 00:06:39.239704 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.239678 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.240012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.239716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsg6\" (UniqueName: \"kubernetes.io/projected/968e1233-5b75-459b-9f7c-4ca46be27371-kube-api-access-crsg6\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.240012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.239751 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.240012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.239801 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/968e1233-5b75-459b-9f7c-4ca46be27371-data-volume\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.240012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.239863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/968e1233-5b75-459b-9f7c-4ca46be27371-crio-socket\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsg6\" (UniqueName: \"kubernetes.io/projected/968e1233-5b75-459b-9f7c-4ca46be27371-kube-api-access-crsg6\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341452 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.341288 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.341452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/968e1233-5b75-459b-9f7c-4ca46be27371-data-volume\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/968e1233-5b75-459b-9f7c-4ca46be27371-crio-socket\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341452 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.341438 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:39.841391147 +0000 UTC m=+238.012424444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.341725 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341444 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/968e1233-5b75-459b-9f7c-4ca46be27371-crio-socket\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341725 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341725 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.341664 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:39.341725 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.341694 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/968e1233-5b75-459b-9f7c-4ca46be27371-data-volume\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.341725 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.341705 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:39.841695014 +0000 UTC m=+238.012728302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : configmap "kube-rbac-proxy" not found Aug 04 00:06:39.350397 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.350376 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsg6\" (UniqueName: \"kubernetes.io/projected/968e1233-5b75-459b-9f7c-4ca46be27371-kube-api-access-crsg6\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.845852 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.845813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.846074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:39.845866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:39.846074 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.845941 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:39.846074 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.845973 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:39.846074 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.846011 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:40.845995859 +0000 UTC m=+239.017029144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : configmap "kube-rbac-proxy" not found Aug 04 00:06:39.846074 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:39.846026 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:40.846020498 +0000 UTC m=+239.017053783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:40.083214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:40.083180 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" event={"ID":"cf63f97c-c35a-4b5d-978a-d6178e0c45f8","Type":"ContainerStarted","Data":"dc7a31f01b8306472adf94d4e24836a1678a75832ef59057587ff1e7f47e6dca"} Aug 04 00:06:40.083416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:40.083232 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" event={"ID":"cf63f97c-c35a-4b5d-978a-d6178e0c45f8","Type":"ContainerStarted","Data":"2b81d3ad1beceb4ce047e6f085c02e3388bf4a948d2de97387d5e0695b50008c"} Aug 04 00:06:40.098920 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:40.098871 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-qxvlm" podStartSLOduration=1.89838766 podStartE2EDuration="3.098854654s" totalImagesPullingTime="1.200466994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:40.098162141 +0000 UTC m=+238.269195451" watchObservedRunningTime="2026-08-04 00:06:40.098854654 +0000 UTC m=+238.269887989" Aug 04 00:06:40.854213 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:40.854168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:40.854213 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:40.854220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:40.854642 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:40.854306 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:40.854642 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:40.854367 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:40.854642 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:40.854391 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.854375412 +0000 UTC m=+241.025408697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : configmap "kube-rbac-proxy" not found Aug 04 00:06:40.854642 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:40.854405 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:42.854398825 +0000 UTC m=+241.025432111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:41.718392 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:41.718356 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kx9jg_539e58ad-eeb7-4a35-8d3f-af0cebbbe186/dns-node-resolver/0.log" Aug 04 00:06:42.064291 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:42.064249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:42.064325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.064412 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.064434 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:50.064416306 +0000 UTC m=+248.235449592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : configmap references non-existent config key: service-ca.crt Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.064463 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs podName:272f48c6-278a-409c-a1e5-a300dcb4a2aa nodeName:}" failed. No retries permitted until 2026-08-04 00:06:50.064450379 +0000 UTC m=+248.235483669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs") pod "router-default-56c65ccbd6-nsg9s" (UID: "272f48c6-278a-409c-a1e5-a300dcb4a2aa") : secret "router-metrics-certs-default" not found Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:42.064495 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.064566 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.064575 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79d7f977cd-44xk6: secret "image-registry-tls" not found Aug 04 00:06:42.064731 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.064602 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls podName:5a88aaff-5987-46b0-9fce-c358a2e75f6f nodeName:}" failed. No retries permitted until 2026-08-04 00:06:50.064595692 +0000 UTC m=+248.235628977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls") pod "image-registry-79d7f977cd-44xk6" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f") : secret "image-registry-tls" not found Aug 04 00:06:42.266230 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:42.266198 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:42.266369 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.266311 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 00:06:42.266369 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.266358 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert podName:788bda68-1682-402e-8cea-39859d7323e3 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:50.266345593 +0000 UTC m=+248.437378879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-whqmh" (UID: "788bda68-1682-402e-8cea-39859d7323e3") : secret "networking-console-plugin-cert" not found Aug 04 00:06:42.871709 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:42.871652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:42.871906 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:42.871788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:42.871906 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.871800 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:42.871906 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.871862 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:46.871847662 +0000 UTC m=+245.042880947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:42.871906 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.871867 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:42.871906 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:42.871909 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:46.871897558 +0000 UTC m=+245.042930843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : configmap "kube-rbac-proxy" not found Aug 04 00:06:43.320043 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:43.320014 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tpvjb_f95614fb-0969-4b83-9db8-7bbc91885d23/node-ca/0.log" Aug 04 00:06:43.919267 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:43.919236 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lb2n9_9b09fb90-1aef-4889-8783-bd14933ff063/serve-healthcheck-canary/0.log" Aug 04 00:06:44.330540 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:44.330510 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-qxvlm_cf63f97c-c35a-4b5d-978a-d6178e0c45f8/migrator/0.log" Aug 04 00:06:44.519670 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:44.519642 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-qxvlm_cf63f97c-c35a-4b5d-978a-d6178e0c45f8/graceful-termination/0.log" Aug 04 00:06:44.719665 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:44.719588 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-z2hl8_a2be9d4e-c830-49a6-91fe-b1d98984dd98/kube-storage-version-migrator-operator/0.log" Aug 04 00:06:46.903340 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:46.903298 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:46.903788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:46.903365 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:46.903788 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:46.903413 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 00:06:46.903788 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:46.903479 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.903464891 +0000 UTC m=+253.074498177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : configmap "kube-rbac-proxy" not found Aug 04 00:06:46.903788 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:46.903482 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 00:06:46.903788 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:46.903552 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls podName:968e1233-5b75-459b-9f7c-4ca46be27371 nodeName:}" failed. No retries permitted until 2026-08-04 00:06:54.903536699 +0000 UTC m=+253.074569984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zw2f8" (UID: "968e1233-5b75-459b-9f7c-4ca46be27371") : secret "insights-runtime-extractor-tls" not found Aug 04 00:06:50.128512 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.128474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:50.129022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.128565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:50.129022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.128589 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:50.129290 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.129269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272f48c6-278a-409c-a1e5-a300dcb4a2aa-service-ca-bundle\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:50.131000 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.130981 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"image-registry-79d7f977cd-44xk6\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:50.131053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.131038 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272f48c6-278a-409c-a1e5-a300dcb4a2aa-metrics-certs\") pod \"router-default-56c65ccbd6-nsg9s\" (UID: \"272f48c6-278a-409c-a1e5-a300dcb4a2aa\") " pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:50.199837 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.199803 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbxcz\"" Aug 04 00:06:50.208572 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.208548 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:50.211811 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.211791 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jh8rb\"" Aug 04 00:06:50.220146 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.220122 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:50.330323 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.330294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:06:50.330510 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:50.330492 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 00:06:50.330592 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:06:50.330580 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert podName:788bda68-1682-402e-8cea-39859d7323e3 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:06.330551938 +0000 UTC m=+264.501585239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-whqmh" (UID: "788bda68-1682-402e-8cea-39859d7323e3") : secret "networking-console-plugin-cert" not found Aug 04 00:06:50.347591 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.347548 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d7f977cd-44xk6"] Aug 04 00:06:50.350331 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:06:50.350302 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a88aaff_5987_46b0_9fce_c358a2e75f6f.slice/crio-b31a66096d49e01b3888ba1c687d00f356b7a842fc6ce6135753252171681264 WatchSource:0}: Error finding container b31a66096d49e01b3888ba1c687d00f356b7a842fc6ce6135753252171681264: Status 404 returned error can't find the container with id b31a66096d49e01b3888ba1c687d00f356b7a842fc6ce6135753252171681264 Aug 04 00:06:50.364050 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:50.364025 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56c65ccbd6-nsg9s"] Aug 04 00:06:50.366856 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:06:50.366832 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272f48c6_278a_409c_a1e5_a300dcb4a2aa.slice/crio-78c5c8c47a70db44220bf76ac815299af989a8cc8b92203cf1c3a4fca074b941 WatchSource:0}: Error finding container 78c5c8c47a70db44220bf76ac815299af989a8cc8b92203cf1c3a4fca074b941: Status 404 returned error can't find the container with id 78c5c8c47a70db44220bf76ac815299af989a8cc8b92203cf1c3a4fca074b941 Aug 04 00:06:51.115883 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.115841 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" event={"ID":"272f48c6-278a-409c-a1e5-a300dcb4a2aa","Type":"ContainerStarted","Data":"7ced58936f94bb978140dae4bf53afef646a444d1c7c57f96cab14ab5c9c78b4"} Aug 04 00:06:51.116121 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.115903 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" event={"ID":"272f48c6-278a-409c-a1e5-a300dcb4a2aa","Type":"ContainerStarted","Data":"78c5c8c47a70db44220bf76ac815299af989a8cc8b92203cf1c3a4fca074b941"} Aug 04 00:06:51.117162 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.117137 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" event={"ID":"5a88aaff-5987-46b0-9fce-c358a2e75f6f","Type":"ContainerStarted","Data":"8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa"} Aug 04 00:06:51.117275 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.117184 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" event={"ID":"5a88aaff-5987-46b0-9fce-c358a2e75f6f","Type":"ContainerStarted","Data":"b31a66096d49e01b3888ba1c687d00f356b7a842fc6ce6135753252171681264"} Aug 04 00:06:51.117275 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.117215 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:06:51.135222 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.135175 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" podStartSLOduration=17.135159969 podStartE2EDuration="17.135159969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:51.134508221 +0000 UTC m=+249.305541529" watchObservedRunningTime="2026-08-04 00:06:51.135159969 +0000 UTC m=+249.306193277" Aug 04 00:06:51.153699 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.153655 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" podStartSLOduration=17.153642194 podStartE2EDuration="17.153642194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:51.153343649 +0000 UTC m=+249.324376947" watchObservedRunningTime="2026-08-04 00:06:51.153642194 +0000 UTC m=+249.324675531" Aug 04 00:06:51.221053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.221013 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:51.223500 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:51.223480 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:52.120715 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:52.120682 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:52.121749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:52.121728 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56c65ccbd6-nsg9s" Aug 04 00:06:54.061674 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.061637 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:06:54.063908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.063877 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba07224-77d2-4138-873b-4278e1b0cbb5-metrics-certs\") pod \"network-metrics-daemon-9wzbx\" (UID: \"3ba07224-77d2-4138-873b-4278e1b0cbb5\") " pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:06:54.073907 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.073890 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9tct\"" Aug 04 00:06:54.082865 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.082845 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9wzbx" Aug 04 00:06:54.209608 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.209578 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9wzbx"] Aug 04 00:06:54.212721 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:06:54.212689 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba07224_77d2_4138_873b_4278e1b0cbb5.slice/crio-6fbe176bd6db12ca9a204b3d57829a81c8685d5f0891652551d5d57040e5b2bc WatchSource:0}: Error finding container 6fbe176bd6db12ca9a204b3d57829a81c8685d5f0891652551d5d57040e5b2bc: Status 404 returned error can't find the container with id 6fbe176bd6db12ca9a204b3d57829a81c8685d5f0891652551d5d57040e5b2bc Aug 04 00:06:54.969143 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.969093 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:54.969311 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.969153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:54.969810 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.969787 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/968e1233-5b75-459b-9f7c-4ca46be27371-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:54.971351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:54.971323 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/968e1233-5b75-459b-9f7c-4ca46be27371-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zw2f8\" (UID: \"968e1233-5b75-459b-9f7c-4ca46be27371\") " pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:55.068377 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:55.068342 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbnpj\"" Aug 04 00:06:55.077072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:55.077045 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zw2f8" Aug 04 00:06:55.131841 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:55.131796 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9wzbx" event={"ID":"3ba07224-77d2-4138-873b-4278e1b0cbb5","Type":"ContainerStarted","Data":"6fbe176bd6db12ca9a204b3d57829a81c8685d5f0891652551d5d57040e5b2bc"} Aug 04 00:06:55.301759 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:55.301726 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zw2f8"] Aug 04 00:06:55.304247 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:06:55.304221 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968e1233_5b75_459b_9f7c_4ca46be27371.slice/crio-073f69dd6674bf4a753fa9c468439cdb8bca2184700797fcaae6b880793ccc9e WatchSource:0}: Error finding container 073f69dd6674bf4a753fa9c468439cdb8bca2184700797fcaae6b880793ccc9e: Status 404 returned error can't find the container with id 073f69dd6674bf4a753fa9c468439cdb8bca2184700797fcaae6b880793ccc9e Aug 04 00:06:56.141834 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:56.141798 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9wzbx" event={"ID":"3ba07224-77d2-4138-873b-4278e1b0cbb5","Type":"ContainerStarted","Data":"79125829ca5e5c43a35c649d3a868f3fbdd31c384058fa7742c1a82d136afa9b"} Aug 04 00:06:56.142316 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:56.141871 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9wzbx" event={"ID":"3ba07224-77d2-4138-873b-4278e1b0cbb5","Type":"ContainerStarted","Data":"529d639f9c884d47cd0d8171cfe84c97859aee88936d5fe10a0e94baf373180a"} Aug 04 00:06:56.143177 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:56.143151 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw2f8" event={"ID":"968e1233-5b75-459b-9f7c-4ca46be27371","Type":"ContainerStarted","Data":"cbce0e33e902d03f2b29a3829fed66bc22af5753b1233c3184f258ffa96c9f22"} Aug 04 00:06:56.143291 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:56.143194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw2f8" event={"ID":"968e1233-5b75-459b-9f7c-4ca46be27371","Type":"ContainerStarted","Data":"073f69dd6674bf4a753fa9c468439cdb8bca2184700797fcaae6b880793ccc9e"} Aug 04 00:06:56.160220 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:56.160182 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9wzbx" podStartSLOduration=253.160909999 podStartE2EDuration="4m14.160166853s" totalImagesPullingTime="999.256854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:56.159701889 +0000 UTC m=+254.330735207" watchObservedRunningTime="2026-08-04 00:06:56.160166853 +0000 UTC m=+254.331200159" Aug 04 00:06:57.149423 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:57.149383 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw2f8" event={"ID":"968e1233-5b75-459b-9f7c-4ca46be27371","Type":"ContainerStarted","Data":"501abaaebbac62fa549385f2fdb19afe3426b89db56f6564f064b00d7775f855"} Aug 04 00:06:58.153956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:58.153922 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zw2f8" event={"ID":"968e1233-5b75-459b-9f7c-4ca46be27371","Type":"ContainerStarted","Data":"2be1e44cb006791435facef94bbbc034ac6e4fa220b6abb2b2b2ff89e2b4c25d"} Aug 04 00:06:58.170955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:06:58.170915 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zw2f8" podStartSLOduration=17.212305501 podStartE2EDuration="19.170902362s" totalImagesPullingTime="1.958596861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 00:06:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:06:58.169910206 +0000 UTC m=+256.340943524" watchObservedRunningTime="2026-08-04 00:06:58.170902362 +0000 UTC m=+256.341935669" Aug 04 00:07:03.424152 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.424116 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d7f977cd-44xk6"] Aug 04 00:07:03.439242 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.439208 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-44zf4"] Aug 04 00:07:03.442563 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.442544 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:03.444839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.444790 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 00:07:03.444839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.444816 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ddx9h\"" Aug 04 00:07:03.445018 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.444856 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 00:07:03.452162 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.452136 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-44zf4"] Aug 04 00:07:03.524097 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.524061 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjp2\" (UniqueName: \"kubernetes.io/projected/af40b08d-2067-4c24-b2ad-d53d0e015a37-kube-api-access-9qjp2\") pod \"downloads-77c974ffcf-44zf4\" (UID: \"af40b08d-2067-4c24-b2ad-d53d0e015a37\") " pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:03.625366 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.625321 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjp2\" (UniqueName: \"kubernetes.io/projected/af40b08d-2067-4c24-b2ad-d53d0e015a37-kube-api-access-9qjp2\") pod \"downloads-77c974ffcf-44zf4\" (UID: \"af40b08d-2067-4c24-b2ad-d53d0e015a37\") " pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:03.637314 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.637288 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjp2\" (UniqueName: \"kubernetes.io/projected/af40b08d-2067-4c24-b2ad-d53d0e015a37-kube-api-access-9qjp2\") pod \"downloads-77c974ffcf-44zf4\" (UID: \"af40b08d-2067-4c24-b2ad-d53d0e015a37\") " pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:03.752282 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.752198 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:03.877184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:03.877146 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-44zf4"] Aug 04 00:07:03.881942 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:03.881917 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf40b08d_2067_4c24_b2ad_d53d0e015a37.slice/crio-4f36756138410509e4a5192d6f6975dc9f1ab910564bb1d0edc2b766199a27e3 WatchSource:0}: Error finding container 4f36756138410509e4a5192d6f6975dc9f1ab910564bb1d0edc2b766199a27e3: Status 404 returned error can't find the container with id 4f36756138410509e4a5192d6f6975dc9f1ab910564bb1d0edc2b766199a27e3 Aug 04 00:07:04.171820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:04.171782 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-44zf4" event={"ID":"af40b08d-2067-4c24-b2ad-d53d0e015a37","Type":"ContainerStarted","Data":"4f36756138410509e4a5192d6f6975dc9f1ab910564bb1d0edc2b766199a27e3"} Aug 04 00:07:06.348419 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:06.348376 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:07:06.351117 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:06.351073 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/788bda68-1682-402e-8cea-39859d7323e3-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-whqmh\" (UID: \"788bda68-1682-402e-8cea-39859d7323e3\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:07:06.582769 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:06.582737 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dxw7k\"" Aug 04 00:07:06.591487 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:06.591469 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" Aug 04 00:07:06.722368 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:06.722333 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-whqmh"] Aug 04 00:07:06.725270 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:06.725235 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788bda68_1682_402e_8cea_39859d7323e3.slice/crio-aca8fac27c26f03a8eac6c7420037ab291f7ea19768d1c3ecd906ccede43e250 WatchSource:0}: Error finding container aca8fac27c26f03a8eac6c7420037ab291f7ea19768d1c3ecd906ccede43e250: Status 404 returned error can't find the container with id aca8fac27c26f03a8eac6c7420037ab291f7ea19768d1c3ecd906ccede43e250 Aug 04 00:07:07.184987 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:07.184915 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" event={"ID":"788bda68-1682-402e-8cea-39859d7323e3","Type":"ContainerStarted","Data":"aca8fac27c26f03a8eac6c7420037ab291f7ea19768d1c3ecd906ccede43e250"} Aug 04 00:07:08.190627 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.190586 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" event={"ID":"788bda68-1682-402e-8cea-39859d7323e3","Type":"ContainerStarted","Data":"90f3a3ec6fbb8ef25fbe764cddad5619a3ca862cc39cf3a8859bf5bab6b17e36"} Aug 04 00:07:08.206379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.206324 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-whqmh" podStartSLOduration=32.935830165 podStartE2EDuration="34.206303386s" totalImagesPullingTime="1.270473221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:08.20522016 +0000 UTC m=+266.376253468" watchObservedRunningTime="2026-08-04 00:07:08.206303386 +0000 UTC m=+266.377336693" Aug 04 00:07:08.402048 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.402012 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-645d98b75f-9hng8"] Aug 04 00:07:08.407082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.407057 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.410214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.410188 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 00:07:08.410355 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.410240 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 00:07:08.410355 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.410307 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m9klf\"" Aug 04 00:07:08.410487 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.410478 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 00:07:08.410747 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.410727 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 00:07:08.410859 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.410836 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 00:07:08.418230 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.418206 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645d98b75f-9hng8"] Aug 04 00:07:08.568818 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.568783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-serving-cert\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.569035 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.568837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-oauth-serving-cert\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.569035 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.568866 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwnr\" (UniqueName: \"kubernetes.io/projected/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-kube-api-access-llwnr\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.569035 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.568892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-config\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.569035 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.568975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-oauth-config\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.569035 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.569012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-service-ca\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.669974 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.669880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-oauth-serving-cert\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.669974 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.669928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwnr\" (UniqueName: \"kubernetes.io/projected/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-kube-api-access-llwnr\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.669974 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.669956 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-config\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.670259 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.670026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-oauth-config\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.670259 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.670144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-service-ca\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.670259 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.670189 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-serving-cert\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.670778 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.670730 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-config\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.670778 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.670730 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-oauth-serving-cert\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.670980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.670959 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-service-ca\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.672913 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.672891 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-oauth-config\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.673184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.673167 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-serving-cert\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.678326 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.678306 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwnr\" (UniqueName: \"kubernetes.io/projected/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-kube-api-access-llwnr\") pod \"console-645d98b75f-9hng8\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.719128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.719083 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:08.867189 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:08.867159 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645d98b75f-9hng8"] Aug 04 00:07:08.871034 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:08.871001 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5176f6d_9f33_4c1d_9743_0f70f95a5d25.slice/crio-dbb3062ead6cf2121d29ca984c8bd5ebe6e8b2dd058e7dfb7d4b09b15da40b0f WatchSource:0}: Error finding container dbb3062ead6cf2121d29ca984c8bd5ebe6e8b2dd058e7dfb7d4b09b15da40b0f: Status 404 returned error can't find the container with id dbb3062ead6cf2121d29ca984c8bd5ebe6e8b2dd058e7dfb7d4b09b15da40b0f Aug 04 00:07:09.195756 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:09.195708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98b75f-9hng8" event={"ID":"b5176f6d-9f33-4c1d-9743-0f70f95a5d25","Type":"ContainerStarted","Data":"dbb3062ead6cf2121d29ca984c8bd5ebe6e8b2dd058e7dfb7d4b09b15da40b0f"} Aug 04 00:07:12.210882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:12.210842 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98b75f-9hng8" event={"ID":"b5176f6d-9f33-4c1d-9743-0f70f95a5d25","Type":"ContainerStarted","Data":"ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a"} Aug 04 00:07:12.231644 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:12.231583 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-645d98b75f-9hng8" podStartSLOduration=1.098502311 podStartE2EDuration="4.23156265s" totalImagesPullingTime="3.133060339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:12.229191984 +0000 UTC m=+270.400225294" watchObservedRunningTime="2026-08-04 00:07:12.23156265 +0000 UTC m=+270.402595960" Aug 04 00:07:13.430182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:13.430123 2571 patch_prober.go:28] interesting pod/image-registry-79d7f977cd-44xk6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 00:07:13.430620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:13.430196 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:07:18.719726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:18.719690 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:18.720184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:18.719754 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:18.721224 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:18.721200 2571 patch_prober.go:28] interesting pod/console-645d98b75f-9hng8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body= Aug 04 00:07:18.721298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:18.721249 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-645d98b75f-9hng8" podUID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" Aug 04 00:07:19.787684 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.787648 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf49c4ccd-89xrq"] Aug 04 00:07:19.790699 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.790678 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.802212 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.801687 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 00:07:19.804605 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.804579 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf49c4ccd-89xrq"] Aug 04 00:07:19.866940 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.866903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-serving-cert\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.867149 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.866964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-oauth-config\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.867149 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.866992 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-service-ca\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.867149 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.867015 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-oauth-serving-cert\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.867149 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.867035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzh2\" (UniqueName: \"kubernetes.io/projected/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-kube-api-access-xdzh2\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.867149 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.867123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-trusted-ca-bundle\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.867356 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.867154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-config\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.967617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.967584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-oauth-config\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.967811 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.967622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-service-ca\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.967811 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.967642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-oauth-serving-cert\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.967811 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.967758 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzh2\" (UniqueName: \"kubernetes.io/projected/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-kube-api-access-xdzh2\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.967982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.967828 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-trusted-ca-bundle\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.967982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.967858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-config\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.968477 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.968117 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-serving-cert\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.968477 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.968423 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-service-ca\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.968477 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.968442 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-oauth-serving-cert\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.968683 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.968670 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-config\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.968776 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.968701 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-trusted-ca-bundle\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.970644 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.970621 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-oauth-config\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.970837 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.970816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-serving-cert\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:19.976052 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:19.976027 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzh2\" (UniqueName: \"kubernetes.io/projected/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-kube-api-access-xdzh2\") pod \"console-7bf49c4ccd-89xrq\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:20.106409 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.106325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:20.238967 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.238938 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf49c4ccd-89xrq"] Aug 04 00:07:20.242090 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:20.242059 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e434d06_1dce_4bdf_9b4c_bfc93cee2d98.slice/crio-c7668570cb0ad3a8c25b55d46598fa525cb191e3c6e7fe2415b5e614fffc9fdc WatchSource:0}: Error finding container c7668570cb0ad3a8c25b55d46598fa525cb191e3c6e7fe2415b5e614fffc9fdc: Status 404 returned error can't find the container with id c7668570cb0ad3a8c25b55d46598fa525cb191e3c6e7fe2415b5e614fffc9fdc Aug 04 00:07:20.242733 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.242710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-44zf4" event={"ID":"af40b08d-2067-4c24-b2ad-d53d0e015a37","Type":"ContainerStarted","Data":"c4aa4e6b0257d15c74f8ee08b5094aa277173229e764a428775d49c6cdcdc339"} Aug 04 00:07:20.242975 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.242950 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:20.244242 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.244222 2571 patch_prober.go:28] interesting pod/downloads-77c974ffcf-44zf4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body= Aug 04 00:07:20.244332 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.244257 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-77c974ffcf-44zf4" podUID="af40b08d-2067-4c24-b2ad-d53d0e015a37" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" Aug 04 00:07:20.260835 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.260786 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-44zf4" podStartSLOduration=1.011518989 podStartE2EDuration="17.260770287s" totalImagesPullingTime="16.249251298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:20.25952749 +0000 UTC m=+278.430560797" watchObservedRunningTime="2026-08-04 00:07:20.260770287 +0000 UTC m=+278.431803593" Aug 04 00:07:20.565868 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.565826 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn"] Aug 04 00:07:20.569092 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.569063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.572356 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572328 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 00:07:20.572492 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572451 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 00:07:20.572573 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572550 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 00:07:20.572681 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572648 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 00:07:20.572867 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj4w2\" (UniqueName: \"kubernetes.io/projected/501ab213-04bd-48c1-be3c-96f343f8b601-kube-api-access-bj4w2\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.572959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572873 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7vqv5\"" Aug 04 00:07:20.572959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.572886 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.573076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.573008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.573076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.573028 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 00:07:20.573076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.573057 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/501ab213-04bd-48c1-be3c-96f343f8b601-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.582801 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.582769 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn"] Aug 04 00:07:20.614841 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.614808 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qb9dn"] Aug 04 00:07:20.617542 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.617521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.619767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.619720 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 00:07:20.619767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.619754 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6k5hv\"" Aug 04 00:07:20.619950 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.619850 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 00:07:20.620087 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.620057 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 00:07:20.673485 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj4w2\" (UniqueName: \"kubernetes.io/projected/501ab213-04bd-48c1-be3c-96f343f8b601-kube-api-access-bj4w2\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.673682 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-root\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.673682 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-wtmp\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.673682 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673570 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-textfile\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.673682 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.673893 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673701 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.673893 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673770 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-sys\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.673893 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.673893 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/501ab213-04bd-48c1-be3c-96f343f8b601-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.674082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673902 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6e8357-0aec-4b62-906a-e471bf0e256c-metrics-client-ca\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.674082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673930 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.674082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-tls\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.674082 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:20.673975 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 04 00:07:20.674082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.673994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcckt\" (UniqueName: \"kubernetes.io/projected/9a6e8357-0aec-4b62-906a-e471bf0e256c-kube-api-access-hcckt\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.674082 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:20.674036 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-tls podName:501ab213-04bd-48c1-be3c-96f343f8b601 nodeName:}" failed. No retries permitted until 2026-08-04 00:07:21.17401671 +0000 UTC m=+279.345050010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-zwbsn" (UID: "501ab213-04bd-48c1-be3c-96f343f8b601") : secret "openshift-state-metrics-tls" not found Aug 04 00:07:20.674635 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.674615 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/501ab213-04bd-48c1-be3c-96f343f8b601-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.676511 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.676489 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.689753 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.689727 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj4w2\" (UniqueName: \"kubernetes.io/projected/501ab213-04bd-48c1-be3c-96f343f8b601-kube-api-access-bj4w2\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:20.774695 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.774650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6e8357-0aec-4b62-906a-e471bf0e256c-metrics-client-ca\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.774695 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.774706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.774952 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.774739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-tls\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.774952 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.774769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcckt\" (UniqueName: \"kubernetes.io/projected/9a6e8357-0aec-4b62-906a-e471bf0e256c-kube-api-access-hcckt\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.774952 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.774814 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-root\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.774952 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.774887 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-root\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.774952 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:20.774934 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 00:07:20.775234 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-wtmp\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775234 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:20.775194 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-tls podName:9a6e8357-0aec-4b62-906a-e471bf0e256c nodeName:}" failed. No retries permitted until 2026-08-04 00:07:21.275173119 +0000 UTC m=+279.446206422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-tls") pod "node-exporter-qb9dn" (UID: "9a6e8357-0aec-4b62-906a-e471bf0e256c") : secret "node-exporter-tls" not found Aug 04 00:07:20.775234 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-textfile\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775397 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775397 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-sys\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775503 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775447 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-sys\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775554 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775190 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-wtmp\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775757 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775715 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6e8357-0aec-4b62-906a-e471bf0e256c-metrics-client-ca\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.775866 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775766 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-textfile\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.776024 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.775991 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.778170 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.778152 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:20.783814 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:20.783791 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcckt\" (UniqueName: \"kubernetes.io/projected/9a6e8357-0aec-4b62-906a-e471bf0e256c-kube-api-access-hcckt\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:21.178594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.178552 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:21.181572 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.181542 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/501ab213-04bd-48c1-be3c-96f343f8b601-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-zwbsn\" (UID: \"501ab213-04bd-48c1-be3c-96f343f8b601\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:21.248354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.248301 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf49c4ccd-89xrq" event={"ID":"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98","Type":"ContainerStarted","Data":"9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd"} Aug 04 00:07:21.248540 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.248368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf49c4ccd-89xrq" event={"ID":"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98","Type":"ContainerStarted","Data":"c7668570cb0ad3a8c25b55d46598fa525cb191e3c6e7fe2415b5e614fffc9fdc"} Aug 04 00:07:21.266047 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.266019 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-44zf4" Aug 04 00:07:21.270955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.270909 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf49c4ccd-89xrq" podStartSLOduration=2.2708914 podStartE2EDuration="2.2708914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:21.269694302 +0000 UTC m=+279.440727633" watchObservedRunningTime="2026-08-04 00:07:21.2708914 +0000 UTC m=+279.441924706" Aug 04 00:07:21.279167 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.279140 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-tls\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:21.282194 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.282169 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a6e8357-0aec-4b62-906a-e471bf0e256c-node-exporter-tls\") pod \"node-exporter-qb9dn\" (UID: \"9a6e8357-0aec-4b62-906a-e471bf0e256c\") " pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:21.480306 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.480221 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" Aug 04 00:07:21.528042 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.528002 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qb9dn" Aug 04 00:07:21.541177 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:21.541137 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6e8357_0aec_4b62_906a_e471bf0e256c.slice/crio-b6ddd9a996f295462391d99884158172e7be4fbb03909f16466dcf1d55c62037 WatchSource:0}: Error finding container b6ddd9a996f295462391d99884158172e7be4fbb03909f16466dcf1d55c62037: Status 404 returned error can't find the container with id b6ddd9a996f295462391d99884158172e7be4fbb03909f16466dcf1d55c62037 Aug 04 00:07:21.640992 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:21.640950 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn"] Aug 04 00:07:21.644087 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:21.644058 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501ab213_04bd_48c1_be3c_96f343f8b601.slice/crio-ede9951fdb78d3eb0cc916c4387d68500d9c1a31be60d9dc98da9aa3992e3ea5 WatchSource:0}: Error finding container ede9951fdb78d3eb0cc916c4387d68500d9c1a31be60d9dc98da9aa3992e3ea5: Status 404 returned error can't find the container with id ede9951fdb78d3eb0cc916c4387d68500d9c1a31be60d9dc98da9aa3992e3ea5 Aug 04 00:07:22.254462 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:22.254423 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" event={"ID":"501ab213-04bd-48c1-be3c-96f343f8b601","Type":"ContainerStarted","Data":"c4cf71752657bf8fbe70cdfb2b738536cc8bfe70075c5688ddaab059e87a323b"} Aug 04 00:07:22.254942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:22.254530 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" event={"ID":"501ab213-04bd-48c1-be3c-96f343f8b601","Type":"ContainerStarted","Data":"294ab7c19f3b64f919da8e87a06b4a27fe616b548889a2ad051c4df342258086"} Aug 04 00:07:22.254942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:22.254566 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" event={"ID":"501ab213-04bd-48c1-be3c-96f343f8b601","Type":"ContainerStarted","Data":"ede9951fdb78d3eb0cc916c4387d68500d9c1a31be60d9dc98da9aa3992e3ea5"} Aug 04 00:07:22.255966 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:22.255923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb9dn" event={"ID":"9a6e8357-0aec-4b62-906a-e471bf0e256c","Type":"ContainerStarted","Data":"b6ddd9a996f295462391d99884158172e7be4fbb03909f16466dcf1d55c62037"} Aug 04 00:07:22.879419 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:22.879325 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-htm5p" podUID="277504f0-4353-4cca-9cc2-3610dbdd9fdc" Aug 04 00:07:23.261288 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.261213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htm5p" Aug 04 00:07:23.430164 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.430127 2571 patch_prober.go:28] interesting pod/image-registry-79d7f977cd-44xk6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 00:07:23.430333 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.430188 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 00:07:23.647743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.647707 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dc5d646c7-fg787"] Aug 04 00:07:23.673812 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.673784 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc5d646c7-fg787"] Aug 04 00:07:23.673936 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.673920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.679309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679285 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 00:07:23.679440 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679330 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 00:07:23.679440 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679331 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 00:07:23.679558 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679454 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mwbrc\"" Aug 04 00:07:23.679619 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679571 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2soijpfj5turh\"" Aug 04 00:07:23.679725 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679706 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 00:07:23.679815 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.679798 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 00:07:23.702752 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.702727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.702869 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.702760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfn8l\" (UniqueName: \"kubernetes.io/projected/752f8113-f595-4dbb-8367-e3573e724f8a-kube-api-access-pfn8l\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.702869 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.702792 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.702996 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.702881 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752f8113-f595-4dbb-8367-e3573e724f8a-metrics-client-ca\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.702996 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.702949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-tls\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.703121 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.702994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.703121 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.703024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-grpc-tls\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.703121 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.703094 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804506 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804481 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-tls\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804555 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-grpc-tls\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804612 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804677 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfn8l\" (UniqueName: \"kubernetes.io/projected/752f8113-f595-4dbb-8367-e3573e724f8a-kube-api-access-pfn8l\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.804882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.804755 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752f8113-f595-4dbb-8367-e3573e724f8a-metrics-client-ca\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.805602 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.805545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752f8113-f595-4dbb-8367-e3573e724f8a-metrics-client-ca\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.807458 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.807409 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-tls\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.807862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.807788 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.808037 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.808013 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-grpc-tls\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.808300 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.808256 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.808300 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.808273 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.808536 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.808515 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/752f8113-f595-4dbb-8367-e3573e724f8a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.813486 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.813463 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfn8l\" (UniqueName: \"kubernetes.io/projected/752f8113-f595-4dbb-8367-e3573e724f8a-kube-api-access-pfn8l\") pod \"thanos-querier-7dc5d646c7-fg787\" (UID: \"752f8113-f595-4dbb-8367-e3573e724f8a\") " pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:23.985053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:23.985010 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:24.152818 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.152782 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc5d646c7-fg787"] Aug 04 00:07:24.156368 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:24.156340 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752f8113_f595_4dbb_8367_e3573e724f8a.slice/crio-c377ce55e49e707d0e57bc15030aab136ba7eb5d5cbedf83a68602dbc86e903a WatchSource:0}: Error finding container c377ce55e49e707d0e57bc15030aab136ba7eb5d5cbedf83a68602dbc86e903a: Status 404 returned error can't find the container with id c377ce55e49e707d0e57bc15030aab136ba7eb5d5cbedf83a68602dbc86e903a Aug 04 00:07:24.267240 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.267145 2571 generic.go:356] "Generic (PLEG): container finished" podID="9a6e8357-0aec-4b62-906a-e471bf0e256c" containerID="deb3d0aedefe016255d4053bc53a49ae2811f80a25fcc086f4510e04880df663" exitCode=0 Aug 04 00:07:24.267703 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.267235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb9dn" event={"ID":"9a6e8357-0aec-4b62-906a-e471bf0e256c","Type":"ContainerDied","Data":"deb3d0aedefe016255d4053bc53a49ae2811f80a25fcc086f4510e04880df663"} Aug 04 00:07:24.269950 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.269915 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" event={"ID":"501ab213-04bd-48c1-be3c-96f343f8b601","Type":"ContainerStarted","Data":"cee5a08c9a4ee8dccda5b51da0ef0c88465ea9ab867643baa057143199240dc7"} Aug 04 00:07:24.271557 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.271523 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"c377ce55e49e707d0e57bc15030aab136ba7eb5d5cbedf83a68602dbc86e903a"} Aug 04 00:07:24.874072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.874000 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-zwbsn" podStartSLOduration=3.002194775 podStartE2EDuration="4.873976082s" totalImagesPullingTime="1.871781307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:24.321228227 +0000 UTC m=+282.492261537" watchObservedRunningTime="2026-08-04 00:07:24.873976082 +0000 UTC m=+283.045009392" Aug 04 00:07:24.875756 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.875731 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dc9c95f58-cr2nt"] Aug 04 00:07:24.894355 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.894067 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc9c95f58-cr2nt"] Aug 04 00:07:24.894355 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.894208 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.897232 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.896990 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 00:07:24.898631 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.898210 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6tsbl\"" Aug 04 00:07:24.898631 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.898459 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-758o4j6v93ld7\"" Aug 04 00:07:24.898631 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.898573 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 00:07:24.899505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.898903 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 00:07:24.899505 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.899205 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 00:07:24.915971 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.915939 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkglx\" (UniqueName: \"kubernetes.io/projected/c978303d-125e-433b-8cae-6532621f590d-kube-api-access-gkglx\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.916237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.915996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c978303d-125e-433b-8cae-6532621f590d-metrics-server-audit-profiles\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.916237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.916050 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-client-ca-bundle\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.916237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.916083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c978303d-125e-433b-8cae-6532621f590d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.916237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.916168 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-secret-metrics-server-client-certs\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.916237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.916225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-secret-metrics-server-tls\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:24.916526 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:24.916250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c978303d-125e-433b-8cae-6532621f590d-audit-log\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.016720 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.016679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-secret-metrics-server-tls\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.016720 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.016720 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c978303d-125e-433b-8cae-6532621f590d-audit-log\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.017008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.016790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkglx\" (UniqueName: \"kubernetes.io/projected/c978303d-125e-433b-8cae-6532621f590d-kube-api-access-gkglx\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.017008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.016926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c978303d-125e-433b-8cae-6532621f590d-metrics-server-audit-profiles\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.017008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.016979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-client-ca-bundle\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.017205 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.017020 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c978303d-125e-433b-8cae-6532621f590d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.017205 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.017073 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-secret-metrics-server-client-certs\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.017307 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.017228 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c978303d-125e-433b-8cae-6532621f590d-audit-log\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.018082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.018054 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c978303d-125e-433b-8cae-6532621f590d-metrics-server-audit-profiles\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.018743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.018672 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c978303d-125e-433b-8cae-6532621f590d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.020214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.020132 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-client-ca-bundle\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.020457 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.020439 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-secret-metrics-server-tls\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.020554 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.020534 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c978303d-125e-433b-8cae-6532621f590d-secret-metrics-server-client-certs\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.025623 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.025590 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkglx\" (UniqueName: \"kubernetes.io/projected/c978303d-125e-433b-8cae-6532621f590d-kube-api-access-gkglx\") pod \"metrics-server-dc9c95f58-cr2nt\" (UID: \"c978303d-125e-433b-8cae-6532621f590d\") " pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.208870 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.208785 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:25.282147 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.282067 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb9dn" event={"ID":"9a6e8357-0aec-4b62-906a-e471bf0e256c","Type":"ContainerStarted","Data":"d778303067bc238ddb65b9062f873b09c72855a81fc149616c72f887f5982b12"} Aug 04 00:07:25.282147 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.282149 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qb9dn" event={"ID":"9a6e8357-0aec-4b62-906a-e471bf0e256c","Type":"ContainerStarted","Data":"4fdb78b3576e1d6dd0fa4d893cc464e14055f7691b2236e38e889114f52eccca"} Aug 04 00:07:25.301331 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.301061 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qb9dn" podStartSLOduration=3.081311243 podStartE2EDuration="5.301040055s" totalImagesPullingTime="2.123202317s" totalInitContainerRuntime="96.526495ms" isStatefulPod=true podCreationTimestamp="2026-08-04 00:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:25.298735193 +0000 UTC m=+283.469768501" watchObservedRunningTime="2026-08-04 00:07:25.301040055 +0000 UTC m=+283.472073363" Aug 04 00:07:25.405609 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:25.405520 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc9c95f58-cr2nt"] Aug 04 00:07:25.409349 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:25.409317 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc978303d_125e_433b_8cae_6532621f590d.slice/crio-f269b9bccebd72200ffd859ddb7d274d620bf198fbbccaedb59072b58006af74 WatchSource:0}: Error finding container f269b9bccebd72200ffd859ddb7d274d620bf198fbbccaedb59072b58006af74: Status 404 returned error can't find the container with id f269b9bccebd72200ffd859ddb7d274d620bf198fbbccaedb59072b58006af74 Aug 04 00:07:26.290814 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:26.290654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" event={"ID":"c978303d-125e-433b-8cae-6532621f590d","Type":"ContainerStarted","Data":"f269b9bccebd72200ffd859ddb7d274d620bf198fbbccaedb59072b58006af74"} Aug 04 00:07:26.636160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:26.636054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:07:26.640120 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:26.640075 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/277504f0-4353-4cca-9cc2-3610dbdd9fdc-metrics-tls\") pod \"dns-default-htm5p\" (UID: \"277504f0-4353-4cca-9cc2-3610dbdd9fdc\") " pod="openshift-dns/dns-default-htm5p" Aug 04 00:07:26.865595 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:26.865549 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7ds4t\"" Aug 04 00:07:26.872274 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:26.872244 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-htm5p" Aug 04 00:07:27.916360 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:27.915402 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-htm5p"] Aug 04 00:07:27.920639 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:27.920604 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277504f0_4353_4cca_9cc2_3610dbdd9fdc.slice/crio-bd871cc42332ad20086f881477cbffaf9d2d7aa974dad7a788dff383367a47f7 WatchSource:0}: Error finding container bd871cc42332ad20086f881477cbffaf9d2d7aa974dad7a788dff383367a47f7: Status 404 returned error can't find the container with id bd871cc42332ad20086f881477cbffaf9d2d7aa974dad7a788dff383367a47f7 Aug 04 00:07:28.301010 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.300968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htm5p" event={"ID":"277504f0-4353-4cca-9cc2-3610dbdd9fdc","Type":"ContainerStarted","Data":"bd871cc42332ad20086f881477cbffaf9d2d7aa974dad7a788dff383367a47f7"} Aug 04 00:07:28.303650 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.303620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"722d31336dd0bf35f63866590f3b87da90deea1f94fa859b28feb599b15c8337"} Aug 04 00:07:28.303784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.303689 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"43a15d5b021c9e2d1f6c1e86116a2c4db6a28a5931c04d7aa0163a5bc6d157d1"} Aug 04 00:07:28.303784 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.303720 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"f24a09dfc6027cfd0d634193015e02ca35218085e819af8193696b31065d6f59"} Aug 04 00:07:28.305366 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.305334 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" event={"ID":"c978303d-125e-433b-8cae-6532621f590d","Type":"ContainerStarted","Data":"a8f7d76745f240820b886932411737113e08e664d36f892455ed525810e5f3cf"} Aug 04 00:07:28.334807 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.334707 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" podStartSLOduration=1.982451532 podStartE2EDuration="4.334685802s" totalImagesPullingTime="2.35223427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:28.333558354 +0000 UTC m=+286.504591664" watchObservedRunningTime="2026-08-04 00:07:28.334685802 +0000 UTC m=+286.505719110" Aug 04 00:07:28.411795 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.411742 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645d98b75f-9hng8"] Aug 04 00:07:28.444251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.444217 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d785578c-5n7dl"] Aug 04 00:07:28.445413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.445354 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerName="registry" containerID="cri-o://8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa" gracePeriod=30 Aug 04 00:07:28.466187 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.466158 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d785578c-5n7dl"] Aug 04 00:07:28.466319 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.466284 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.552753 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.552659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-service-ca\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.552753 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.552725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-console-config\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.552969 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.552761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-trusted-ca-bundle\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.552969 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.552791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-serving-cert\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.552969 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.552893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-oauth-config\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.552969 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.552941 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9rk\" (UniqueName: \"kubernetes.io/projected/b0265d68-2f39-4db6-b858-6374a04012f4-kube-api-access-gq9rk\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.553184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.553001 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-oauth-serving-cert\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.653720 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-service-ca\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.653882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-console-config\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.653882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653798 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-trusted-ca-bundle\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.653882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-serving-cert\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.653882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-oauth-config\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.654092 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9rk\" (UniqueName: \"kubernetes.io/projected/b0265d68-2f39-4db6-b858-6374a04012f4-kube-api-access-gq9rk\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.654092 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.653937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-oauth-serving-cert\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.654677 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.654654 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-oauth-serving-cert\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.655126 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.654772 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-console-config\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.655746 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.655708 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-service-ca\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.656400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.655892 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-trusted-ca-bundle\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.658366 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.658325 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-oauth-config\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.660123 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.660074 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-serving-cert\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.663862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.663841 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9rk\" (UniqueName: \"kubernetes.io/projected/b0265d68-2f39-4db6-b858-6374a04012f4-kube-api-access-gq9rk\") pod \"console-57d785578c-5n7dl\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.724820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.724766 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:07:28.754562 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754363 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.754722 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754641 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-certificates\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.754722 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754675 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-image-registry-private-configuration\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.754722 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754698 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw9l9\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-kube-api-access-xw9l9\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.754880 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754761 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a88aaff-5987-46b0-9fce-c358a2e75f6f-ca-trust-extracted\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.754880 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754799 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-trusted-ca\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.754880 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754857 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-installation-pull-secrets\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.755016 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.754894 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-bound-sa-token\") pod \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\" (UID: \"5a88aaff-5987-46b0-9fce-c358a2e75f6f\") " Aug 04 00:07:28.756706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.756232 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:07:28.756706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.756561 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:07:28.761778 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.761537 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:07:28.767593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.767550 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:07:28.768030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.767979 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-kube-api-access-xw9l9" (OuterVolumeSpecName: "kube-api-access-xw9l9") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "kube-api-access-xw9l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:07:28.768326 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.768291 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:07:28.768477 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.768438 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a88aaff-5987-46b0-9fce-c358a2e75f6f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:07:28.770296 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.770269 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5a88aaff-5987-46b0-9fce-c358a2e75f6f" (UID: "5a88aaff-5987-46b0-9fce-c358a2e75f6f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:07:28.777581 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.777555 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856302 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856327 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-registry-certificates\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856338 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-image-registry-private-configuration\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856347 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw9l9\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-kube-api-access-xw9l9\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856361 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5a88aaff-5987-46b0-9fce-c358a2e75f6f-ca-trust-extracted\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856375 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a88aaff-5987-46b0-9fce-c358a2e75f6f-trusted-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856388 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5a88aaff-5987-46b0-9fce-c358a2e75f6f-installation-pull-secrets\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.856388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.856398 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a88aaff-5987-46b0-9fce-c358a2e75f6f-bound-sa-token\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:28.979548 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:28.979512 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d785578c-5n7dl"] Aug 04 00:07:28.984450 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:28.984415 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0265d68_2f39_4db6_b858_6374a04012f4.slice/crio-032e4135ec31d8a13b38286b9bc5fb65767381f0b1ee935d16e606548f506674 WatchSource:0}: Error finding container 032e4135ec31d8a13b38286b9bc5fb65767381f0b1ee935d16e606548f506674: Status 404 returned error can't find the container with id 032e4135ec31d8a13b38286b9bc5fb65767381f0b1ee935d16e606548f506674 Aug 04 00:07:29.314521 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.314471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d785578c-5n7dl" event={"ID":"b0265d68-2f39-4db6-b858-6374a04012f4","Type":"ContainerStarted","Data":"3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe"} Aug 04 00:07:29.314697 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.314575 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d785578c-5n7dl" event={"ID":"b0265d68-2f39-4db6-b858-6374a04012f4","Type":"ContainerStarted","Data":"032e4135ec31d8a13b38286b9bc5fb65767381f0b1ee935d16e606548f506674"} Aug 04 00:07:29.316653 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.316616 2571 generic.go:356] "Generic (PLEG): container finished" podID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerID="8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa" exitCode=0 Aug 04 00:07:29.316899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.316779 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" event={"ID":"5a88aaff-5987-46b0-9fce-c358a2e75f6f","Type":"ContainerDied","Data":"8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa"} Aug 04 00:07:29.316980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.316893 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" Aug 04 00:07:29.316980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.316949 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d7f977cd-44xk6" event={"ID":"5a88aaff-5987-46b0-9fce-c358a2e75f6f","Type":"ContainerDied","Data":"b31a66096d49e01b3888ba1c687d00f356b7a842fc6ce6135753252171681264"} Aug 04 00:07:29.317080 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.316984 2571 scope.go:117] "RemoveContainer" containerID="8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa" Aug 04 00:07:29.336128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.335679 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57d785578c-5n7dl" podStartSLOduration=1.335658397 podStartE2EDuration="1.335658397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:29.334487921 +0000 UTC m=+287.505521235" watchObservedRunningTime="2026-08-04 00:07:29.335658397 +0000 UTC m=+287.506691705" Aug 04 00:07:29.347847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.347819 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79d7f977cd-44xk6"] Aug 04 00:07:29.351132 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.351073 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79d7f977cd-44xk6"] Aug 04 00:07:29.459472 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.459446 2571 scope.go:117] "RemoveContainer" containerID="8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa" Aug 04 00:07:29.459915 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:29.459886 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa\": container with ID starting with 8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa not found: ID does not exist" containerID="8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa" Aug 04 00:07:29.460025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:29.459929 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa"} err="failed to get container status \"8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa\": rpc error: code = NotFound desc = could not find container \"8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa\": container with ID starting with 8c8d4d9f55950e251d92750dd6efaae04d24365c44fcf461478ad6430f8b60fa not found: ID does not exist" Aug 04 00:07:30.106489 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:30.106448 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:30.106935 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:30.106516 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:07:30.108093 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:30.108067 2571 patch_prober.go:28] interesting pod/console-7bf49c4ccd-89xrq container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused" start-of-body= Aug 04 00:07:30.108201 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:30.108127 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7bf49c4ccd-89xrq" podUID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" containerName="console" probeResult="failure" output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused" Aug 04 00:07:30.376747 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:30.376665 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" path="/var/lib/kubelet/pods/5a88aaff-5987-46b0-9fce-c358a2e75f6f/volumes" Aug 04 00:07:31.331228 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.331127 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"daab07314f7b66461ccfbd98ba2fdc0d8f92b03e73121c7c56f92b20879743b5"} Aug 04 00:07:31.331228 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.331202 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"aecdb3707bd435114311d3b54380d334fad1ce1e2c37aa39b29dc4693aa4f966"} Aug 04 00:07:31.331698 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.331249 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:31.331698 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.331289 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" event={"ID":"752f8113-f595-4dbb-8367-e3573e724f8a","Type":"ContainerStarted","Data":"4619b9a6e4bb39a998004457ee7eec223a7b740290904f2ecbad18fd575f3d9c"} Aug 04 00:07:31.332748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.332724 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htm5p" event={"ID":"277504f0-4353-4cca-9cc2-3610dbdd9fdc","Type":"ContainerStarted","Data":"f066e7b601ad2c1d900a2fccf5ffe3416508361666fe31cca9d599fcdde551e4"} Aug 04 00:07:31.332840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.332767 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-htm5p" event={"ID":"277504f0-4353-4cca-9cc2-3610dbdd9fdc","Type":"ContainerStarted","Data":"01372438e543167f34bde41ce3f3f078d12b93a1ba632d361d2cbd386fb35080"} Aug 04 00:07:31.332840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.332790 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-htm5p" Aug 04 00:07:31.355763 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.355709 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" podStartSLOduration=2.54616174 podStartE2EDuration="8.355688838s" totalImagesPullingTime="5.809527098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:31.353344018 +0000 UTC m=+289.524377324" watchObservedRunningTime="2026-08-04 00:07:31.355688838 +0000 UTC m=+289.526722146" Aug 04 00:07:31.368825 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:31.368780 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-htm5p" podStartSLOduration=252.513457594 podStartE2EDuration="4m15.368765602s" totalImagesPullingTime="2.855308008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:31.367846427 +0000 UTC m=+289.538879734" watchObservedRunningTime="2026-08-04 00:07:31.368765602 +0000 UTC m=+289.539798909" Aug 04 00:07:34.999074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:34.999039 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf49c4ccd-89xrq"] Aug 04 00:07:35.031711 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.031678 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-848df65887-xxjzk"] Aug 04 00:07:35.031979 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.031967 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerName="registry" Aug 04 00:07:35.032022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.031981 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerName="registry" Aug 04 00:07:35.032055 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.032036 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a88aaff-5987-46b0-9fce-c358a2e75f6f" containerName="registry" Aug 04 00:07:35.049315 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.049291 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848df65887-xxjzk"] Aug 04 00:07:35.049452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.049388 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111171 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111143 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-oauth-config\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7pg\" (UniqueName: \"kubernetes.io/projected/a21d8de9-13da-4355-9833-48813465d31e-kube-api-access-9c7pg\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-serving-cert\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111260 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-console-config\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111328 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-trusted-ca-bundle\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111499 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111358 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-oauth-serving-cert\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.111499 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.111386 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-service-ca\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212789 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212755 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-oauth-config\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7pg\" (UniqueName: \"kubernetes.io/projected/a21d8de9-13da-4355-9833-48813465d31e-kube-api-access-9c7pg\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-serving-cert\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212849 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-console-config\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-trusted-ca-bundle\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-oauth-serving-cert\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.212956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.212939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-service-ca\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.213783 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.213752 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-console-config\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.213888 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.213785 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-oauth-serving-cert\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.213888 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.213759 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-service-ca\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.214047 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.214017 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-trusted-ca-bundle\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.215309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.215286 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-serving-cert\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.215405 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.215362 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-oauth-config\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.220341 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.220314 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7pg\" (UniqueName: \"kubernetes.io/projected/a21d8de9-13da-4355-9833-48813465d31e-kube-api-access-9c7pg\") pod \"console-848df65887-xxjzk\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.358169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.358137 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:35.506304 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:35.506280 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848df65887-xxjzk"] Aug 04 00:07:35.508759 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:07:35.508724 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21d8de9_13da_4355_9833_48813465d31e.slice/crio-23a92585fe83b3a0eb14e4a22b696e2057a47c124b5cbd5a4c059fd475471e1e WatchSource:0}: Error finding container 23a92585fe83b3a0eb14e4a22b696e2057a47c124b5cbd5a4c059fd475471e1e: Status 404 returned error can't find the container with id 23a92585fe83b3a0eb14e4a22b696e2057a47c124b5cbd5a4c059fd475471e1e Aug 04 00:07:36.353292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:36.353249 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848df65887-xxjzk" event={"ID":"a21d8de9-13da-4355-9833-48813465d31e","Type":"ContainerStarted","Data":"c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce"} Aug 04 00:07:36.353702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:36.353309 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848df65887-xxjzk" event={"ID":"a21d8de9-13da-4355-9833-48813465d31e","Type":"ContainerStarted","Data":"23a92585fe83b3a0eb14e4a22b696e2057a47c124b5cbd5a4c059fd475471e1e"} Aug 04 00:07:36.370638 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:36.370586 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-848df65887-xxjzk" podStartSLOduration=1.370571537 podStartE2EDuration="1.370571537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:07:36.368932937 +0000 UTC m=+294.539966243" watchObservedRunningTime="2026-08-04 00:07:36.370571537 +0000 UTC m=+294.541604844" Aug 04 00:07:37.342702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:37.342677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dc5d646c7-fg787" Aug 04 00:07:38.778496 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:38.778467 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:38.778960 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:38.778526 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:38.784377 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:38.784356 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:39.367767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:39.367739 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:07:41.339153 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:41.339121 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-htm5p" Aug 04 00:07:42.249832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:42.249809 2571 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 00:07:45.208934 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.208896 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:45.209401 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.208958 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:07:45.358981 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.358946 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:45.359193 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.359008 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:45.363822 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.363798 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:45.389529 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.389506 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:07:45.434717 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:45.434681 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d785578c-5n7dl"] Aug 04 00:07:50.350238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:50.350211 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-htm5p_277504f0-4353-4cca-9cc2-3610dbdd9fdc/dns/0.log" Aug 04 00:07:50.550692 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:50.550663 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-htm5p_277504f0-4353-4cca-9cc2-3610dbdd9fdc/kube-rbac-proxy/0.log" Aug 04 00:07:50.750341 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:50.750272 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kx9jg_539e58ad-eeb7-4a35-8d3f-af0cebbbe186/dns-node-resolver/0.log" Aug 04 00:07:51.351839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:51.351804 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56c65ccbd6-nsg9s_272f48c6-278a-409c-a1e5-a300dcb4a2aa/router/0.log" Aug 04 00:07:51.750173 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:51.750077 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lb2n9_9b09fb90-1aef-4889-8783-bd14933ff063/serve-healthcheck-canary/0.log" Aug 04 00:07:53.436705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.436656 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-645d98b75f-9hng8" podUID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" containerName="console" containerID="cri-o://ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a" gracePeriod=15 Aug 04 00:07:53.713630 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.713558 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:53.885755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.885723 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-oauth-config\") pod \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " Aug 04 00:07:53.885942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.885779 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-service-ca\") pod \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " Aug 04 00:07:53.885942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.885810 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-config\") pod \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " Aug 04 00:07:53.885942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.885835 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-serving-cert\") pod \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " Aug 04 00:07:53.885942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.885905 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llwnr\" (UniqueName: \"kubernetes.io/projected/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-kube-api-access-llwnr\") pod \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " Aug 04 00:07:53.886183 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.885960 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-oauth-serving-cert\") pod \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\" (UID: \"b5176f6d-9f33-4c1d-9743-0f70f95a5d25\") " Aug 04 00:07:53.886399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.886351 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5176f6d-9f33-4c1d-9743-0f70f95a5d25" (UID: "b5176f6d-9f33-4c1d-9743-0f70f95a5d25"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:07:53.886529 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.886363 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-config" (OuterVolumeSpecName: "console-config") pod "b5176f6d-9f33-4c1d-9743-0f70f95a5d25" (UID: "b5176f6d-9f33-4c1d-9743-0f70f95a5d25"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:07:53.886529 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.886483 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5176f6d-9f33-4c1d-9743-0f70f95a5d25" (UID: "b5176f6d-9f33-4c1d-9743-0f70f95a5d25"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:07:53.888112 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.888076 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5176f6d-9f33-4c1d-9743-0f70f95a5d25" (UID: "b5176f6d-9f33-4c1d-9743-0f70f95a5d25"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:07:53.888186 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.888146 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-kube-api-access-llwnr" (OuterVolumeSpecName: "kube-api-access-llwnr") pod "b5176f6d-9f33-4c1d-9743-0f70f95a5d25" (UID: "b5176f6d-9f33-4c1d-9743-0f70f95a5d25"). InnerVolumeSpecName "kube-api-access-llwnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:07:53.888280 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.888262 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5176f6d-9f33-4c1d-9743-0f70f95a5d25" (UID: "b5176f6d-9f33-4c1d-9743-0f70f95a5d25"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:07:53.986735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.986644 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-oauth-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:53.986735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.986684 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:53.986735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.986700 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:53.986735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.986713 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-console-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:53.986735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.986729 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llwnr\" (UniqueName: \"kubernetes.io/projected/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-kube-api-access-llwnr\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:53.986988 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:53.986742 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5176f6d-9f33-4c1d-9743-0f70f95a5d25-oauth-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:07:54.425539 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.425508 2571 generic.go:356] "Generic (PLEG): container finished" podID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" containerID="ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a" exitCode=0 Aug 04 00:07:54.425694 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.425555 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98b75f-9hng8" Aug 04 00:07:54.425694 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.425587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98b75f-9hng8" event={"ID":"b5176f6d-9f33-4c1d-9743-0f70f95a5d25","Type":"ContainerDied","Data":"ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a"} Aug 04 00:07:54.425694 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.425632 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98b75f-9hng8" event={"ID":"b5176f6d-9f33-4c1d-9743-0f70f95a5d25","Type":"ContainerDied","Data":"dbb3062ead6cf2121d29ca984c8bd5ebe6e8b2dd058e7dfb7d4b09b15da40b0f"} Aug 04 00:07:54.425694 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.425658 2571 scope.go:117] "RemoveContainer" containerID="ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a" Aug 04 00:07:54.433667 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.433653 2571 scope.go:117] "RemoveContainer" containerID="ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a" Aug 04 00:07:54.433924 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:07:54.433908 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a\": container with ID starting with ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a not found: ID does not exist" containerID="ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a" Aug 04 00:07:54.434003 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.433930 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a"} err="failed to get container status \"ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a\": rpc error: code = NotFound desc = could not find container \"ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a\": container with ID starting with ea0f2ed77cf1cedeebade95c2c26a99b529389961f2f901f2fdacda1973fac4a not found: ID does not exist" Aug 04 00:07:54.440813 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.440793 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645d98b75f-9hng8"] Aug 04 00:07:54.444346 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:54.444329 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-645d98b75f-9hng8"] Aug 04 00:07:56.375905 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:07:56.375871 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" path="/var/lib/kubelet/pods/b5176f6d-9f33-4c1d-9743-0f70f95a5d25/volumes" Aug 04 00:08:00.017447 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.017402 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf49c4ccd-89xrq" podUID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" containerName="console" containerID="cri-o://9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd" gracePeriod=15 Aug 04 00:08:00.298921 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.298889 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:08:00.338677 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338638 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-trusted-ca-bundle\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.338857 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338698 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-oauth-config\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.338857 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338751 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-oauth-serving-cert\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.338857 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338801 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-config\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.338857 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338844 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdzh2\" (UniqueName: \"kubernetes.io/projected/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-kube-api-access-xdzh2\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.339067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338883 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-service-ca\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.339067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338916 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-serving-cert\") pod \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\" (UID: \"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98\") " Aug 04 00:08:00.339067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.338986 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:00.339252 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.339186 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-trusted-ca-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.339252 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.339230 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:00.339473 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.339443 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-service-ca" (OuterVolumeSpecName: "service-ca") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:00.339580 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.339555 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-config" (OuterVolumeSpecName: "console-config") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:00.340923 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.340894 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:08:00.341134 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.341088 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-kube-api-access-xdzh2" (OuterVolumeSpecName: "kube-api-access-xdzh2") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "kube-api-access-xdzh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:08:00.341655 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.341632 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" (UID: "0e434d06-1dce-4bdf-9b4c-bfc93cee2d98"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:08:00.440595 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.440559 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-oauth-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.440595 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.440597 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-oauth-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.440797 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.440612 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.440797 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.440626 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdzh2\" (UniqueName: \"kubernetes.io/projected/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-kube-api-access-xdzh2\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.440797 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.440639 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.440797 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.440652 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98-console-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:00.451025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.450994 2571 generic.go:356] "Generic (PLEG): container finished" podID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" containerID="9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd" exitCode=0 Aug 04 00:08:00.451208 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.451060 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf49c4ccd-89xrq" Aug 04 00:08:00.451208 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.451081 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf49c4ccd-89xrq" event={"ID":"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98","Type":"ContainerDied","Data":"9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd"} Aug 04 00:08:00.451208 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.451147 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf49c4ccd-89xrq" event={"ID":"0e434d06-1dce-4bdf-9b4c-bfc93cee2d98","Type":"ContainerDied","Data":"c7668570cb0ad3a8c25b55d46598fa525cb191e3c6e7fe2415b5e614fffc9fdc"} Aug 04 00:08:00.451208 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.451169 2571 scope.go:117] "RemoveContainer" containerID="9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd" Aug 04 00:08:00.460377 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.460345 2571 scope.go:117] "RemoveContainer" containerID="9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd" Aug 04 00:08:00.460777 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:08:00.460751 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd\": container with ID starting with 9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd not found: ID does not exist" containerID="9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd" Aug 04 00:08:00.460850 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.460790 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd"} err="failed to get container status \"9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd\": rpc error: code = NotFound desc = could not find container \"9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd\": container with ID starting with 9c8a836247457f815bcc66c63c7703a281816a8c63c4366ae2526e034ef5c6dd not found: ID does not exist" Aug 04 00:08:00.466226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.466199 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf49c4ccd-89xrq"] Aug 04 00:08:00.470566 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:00.470538 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf49c4ccd-89xrq"] Aug 04 00:08:02.376191 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:02.376159 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" path="/var/lib/kubelet/pods/0e434d06-1dce-4bdf-9b4c-bfc93cee2d98/volumes" Aug 04 00:08:02.465711 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:02.465676 2571 generic.go:356] "Generic (PLEG): container finished" podID="a2be9d4e-c830-49a6-91fe-b1d98984dd98" containerID="dcb4352eea7b0298504ea4d71c68f33592aa53fb30a08950fec934652c45e993" exitCode=0 Aug 04 00:08:02.465872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:02.465744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" event={"ID":"a2be9d4e-c830-49a6-91fe-b1d98984dd98","Type":"ContainerDied","Data":"dcb4352eea7b0298504ea4d71c68f33592aa53fb30a08950fec934652c45e993"} Aug 04 00:08:02.466081 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:02.466064 2571 scope.go:117] "RemoveContainer" containerID="dcb4352eea7b0298504ea4d71c68f33592aa53fb30a08950fec934652c45e993" Aug 04 00:08:03.471027 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:03.470993 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-z2hl8" event={"ID":"a2be9d4e-c830-49a6-91fe-b1d98984dd98","Type":"ContainerStarted","Data":"5588993b5142dc4958cdacf0bcd03501c47f2e0c5ce42db3e286b2b2623d47ff"} Aug 04 00:08:05.214218 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:05.214187 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:08:05.218266 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:05.218242 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dc9c95f58-cr2nt" Aug 04 00:08:10.458040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.458000 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57d785578c-5n7dl" podUID="b0265d68-2f39-4db6-b858-6374a04012f4" containerName="console" containerID="cri-o://3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe" gracePeriod=15 Aug 04 00:08:10.695464 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.695439 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:08:10.722733 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722647 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq9rk\" (UniqueName: \"kubernetes.io/projected/b0265d68-2f39-4db6-b858-6374a04012f4-kube-api-access-gq9rk\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.722733 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722722 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-service-ca\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.722959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722818 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-trusted-ca-bundle\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.722959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722864 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-console-config\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.722959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722901 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-serving-cert\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.722959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722939 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-oauth-serving-cert\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.723211 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.722970 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-oauth-config\") pod \"b0265d68-2f39-4db6-b858-6374a04012f4\" (UID: \"b0265d68-2f39-4db6-b858-6374a04012f4\") " Aug 04 00:08:10.723211 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.723170 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:10.723211 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.723178 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:10.723372 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.723252 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:10.723372 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.723268 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-trusted-ca-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:10.723372 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.723290 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-console-config" (OuterVolumeSpecName: "console-config") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:10.723372 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.723324 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:08:10.725079 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.725054 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:08:10.725236 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.725117 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:08:10.725328 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.725303 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0265d68-2f39-4db6-b858-6374a04012f4-kube-api-access-gq9rk" (OuterVolumeSpecName: "kube-api-access-gq9rk") pod "b0265d68-2f39-4db6-b858-6374a04012f4" (UID: "b0265d68-2f39-4db6-b858-6374a04012f4"). InnerVolumeSpecName "kube-api-access-gq9rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:08:10.824553 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.824517 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-console-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:10.824553 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.824546 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:10.824553 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.824556 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b0265d68-2f39-4db6-b858-6374a04012f4-oauth-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:10.824786 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.824565 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b0265d68-2f39-4db6-b858-6374a04012f4-console-oauth-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:10.824786 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:10.824575 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9rk\" (UniqueName: \"kubernetes.io/projected/b0265d68-2f39-4db6-b858-6374a04012f4-kube-api-access-gq9rk\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:08:11.503156 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.503119 2571 generic.go:356] "Generic (PLEG): container finished" podID="b0265d68-2f39-4db6-b858-6374a04012f4" containerID="3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe" exitCode=0 Aug 04 00:08:11.503156 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.503159 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d785578c-5n7dl" Aug 04 00:08:11.503651 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.503165 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d785578c-5n7dl" event={"ID":"b0265d68-2f39-4db6-b858-6374a04012f4","Type":"ContainerDied","Data":"3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe"} Aug 04 00:08:11.503651 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.503211 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d785578c-5n7dl" event={"ID":"b0265d68-2f39-4db6-b858-6374a04012f4","Type":"ContainerDied","Data":"032e4135ec31d8a13b38286b9bc5fb65767381f0b1ee935d16e606548f506674"} Aug 04 00:08:11.503651 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.503238 2571 scope.go:117] "RemoveContainer" containerID="3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe" Aug 04 00:08:11.511922 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.511898 2571 scope.go:117] "RemoveContainer" containerID="3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe" Aug 04 00:08:11.512179 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:08:11.512161 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe\": container with ID starting with 3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe not found: ID does not exist" containerID="3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe" Aug 04 00:08:11.512237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.512189 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe"} err="failed to get container status \"3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe\": rpc error: code = NotFound desc = could not find container \"3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe\": container with ID starting with 3e3e0faf6830f9765e2fdba6a959aa7a7d78e6c2f4733bf1df120ef3ed870ffe not found: ID does not exist" Aug 04 00:08:11.524261 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.524237 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d785578c-5n7dl"] Aug 04 00:08:11.530189 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:11.530169 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d785578c-5n7dl"] Aug 04 00:08:12.375471 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:12.375426 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0265d68-2f39-4db6-b858-6374a04012f4" path="/var/lib/kubelet/pods/b0265d68-2f39-4db6-b858-6374a04012f4/volumes" Aug 04 00:08:56.894932 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.894862 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57cb95b47-jxk2w"] Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895171 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895184 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895198 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895204 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895213 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0265d68-2f39-4db6-b858-6374a04012f4" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895218 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0265d68-2f39-4db6-b858-6374a04012f4" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895259 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e434d06-1dce-4bdf-9b4c-bfc93cee2d98" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895271 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5176f6d-9f33-4c1d-9743-0f70f95a5d25" containerName="console" Aug 04 00:08:56.895427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.895279 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0265d68-2f39-4db6-b858-6374a04012f4" containerName="console" Aug 04 00:08:56.897945 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.897924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:56.907879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:56.907858 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57cb95b47-jxk2w"] Aug 04 00:08:57.074771 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.074732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-oauth-config\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.074771 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.074774 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zdh\" (UniqueName: \"kubernetes.io/projected/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-kube-api-access-49zdh\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.075041 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.074843 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-oauth-serving-cert\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.075041 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.074919 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-config\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.075041 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.074972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-trusted-ca-bundle\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.075041 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.075005 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-serving-cert\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.075226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.075060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-service-ca\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176085 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.175976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-trusted-ca-bundle\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176085 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.176030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-serving-cert\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176085 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.176058 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-service-ca\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.176135 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-oauth-config\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.176168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zdh\" (UniqueName: \"kubernetes.io/projected/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-kube-api-access-49zdh\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.176216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-oauth-serving-cert\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.176399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.176250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-config\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.182853 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.177152 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-oauth-serving-cert\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.182853 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.177292 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-config\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.182853 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.177607 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-service-ca\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.182853 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.179571 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-oauth-config\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.182853 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.179700 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-serving-cert\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.184266 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.184242 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zdh\" (UniqueName: \"kubernetes.io/projected/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-kube-api-access-49zdh\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.184366 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.184351 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-trusted-ca-bundle\") pod \"console-57cb95b47-jxk2w\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.207328 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.207300 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:08:57.337236 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.337212 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57cb95b47-jxk2w"] Aug 04 00:08:57.339479 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:08:57.339447 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345570f1_6d6d_4b0c_b7d7_d8d1a6cea15a.slice/crio-c777d93db46300e150b6657be5bd027857fe09cb37273ac8e1e4785c4b05ea95 WatchSource:0}: Error finding container c777d93db46300e150b6657be5bd027857fe09cb37273ac8e1e4785c4b05ea95: Status 404 returned error can't find the container with id c777d93db46300e150b6657be5bd027857fe09cb37273ac8e1e4785c4b05ea95 Aug 04 00:08:57.674985 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.674943 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb95b47-jxk2w" event={"ID":"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a","Type":"ContainerStarted","Data":"4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0"} Aug 04 00:08:57.675231 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.675006 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb95b47-jxk2w" event={"ID":"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a","Type":"ContainerStarted","Data":"c777d93db46300e150b6657be5bd027857fe09cb37273ac8e1e4785c4b05ea95"} Aug 04 00:08:57.692862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:08:57.692811 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57cb95b47-jxk2w" podStartSLOduration=1.6927963369999999 podStartE2EDuration="1.692796337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:08:57.691665053 +0000 UTC m=+375.862698360" watchObservedRunningTime="2026-08-04 00:08:57.692796337 +0000 UTC m=+375.863829644" Aug 04 00:09:07.208244 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:07.208192 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:09:07.208840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:07.208293 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:09:07.213138 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:07.213114 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:09:07.715735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:07.715699 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:09:07.761702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:07.761674 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848df65887-xxjzk"] Aug 04 00:09:32.781195 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:32.781154 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-848df65887-xxjzk" podUID="a21d8de9-13da-4355-9833-48813465d31e" containerName="console" containerID="cri-o://c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce" gracePeriod=15 Aug 04 00:09:33.015848 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.015822 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:09:33.048015 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.047945 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-service-ca\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048015 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.047997 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-serving-cert\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048040 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-console-config\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048079 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-oauth-serving-cert\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048134 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-oauth-config\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048176 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-trusted-ca-bundle\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048217 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c7pg\" (UniqueName: \"kubernetes.io/projected/a21d8de9-13da-4355-9833-48813465d31e-kube-api-access-9c7pg\") pod \"a21d8de9-13da-4355-9833-48813465d31e\" (UID: \"a21d8de9-13da-4355-9833-48813465d31e\") " Aug 04 00:09:33.048482 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048401 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-service-ca" (OuterVolumeSpecName: "service-ca") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:09:33.048482 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048452 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:09:33.048594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048475 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-console-config" (OuterVolumeSpecName: "console-config") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:09:33.048815 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.048790 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:09:33.050306 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.050279 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:09:33.050405 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.050304 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:09:33.050768 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.050748 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21d8de9-13da-4355-9833-48813465d31e-kube-api-access-9c7pg" (OuterVolumeSpecName: "kube-api-access-9c7pg") pod "a21d8de9-13da-4355-9833-48813465d31e" (UID: "a21d8de9-13da-4355-9833-48813465d31e"). InnerVolumeSpecName "kube-api-access-9c7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:09:33.149495 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149444 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.149495 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149487 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.149495 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149499 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-console-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.149495 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149508 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-oauth-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.149495 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149517 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a21d8de9-13da-4355-9833-48813465d31e-console-oauth-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.149800 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149526 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a21d8de9-13da-4355-9833-48813465d31e-trusted-ca-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.149800 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.149534 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7pg\" (UniqueName: \"kubernetes.io/projected/a21d8de9-13da-4355-9833-48813465d31e-kube-api-access-9c7pg\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:09:33.811945 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.811912 2571 generic.go:356] "Generic (PLEG): container finished" podID="a21d8de9-13da-4355-9833-48813465d31e" containerID="c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce" exitCode=0 Aug 04 00:09:33.812357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.811975 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848df65887-xxjzk" Aug 04 00:09:33.812357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.812003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848df65887-xxjzk" event={"ID":"a21d8de9-13da-4355-9833-48813465d31e","Type":"ContainerDied","Data":"c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce"} Aug 04 00:09:33.812357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.812076 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848df65887-xxjzk" event={"ID":"a21d8de9-13da-4355-9833-48813465d31e","Type":"ContainerDied","Data":"23a92585fe83b3a0eb14e4a22b696e2057a47c124b5cbd5a4c059fd475471e1e"} Aug 04 00:09:33.812357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.812096 2571 scope.go:117] "RemoveContainer" containerID="c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce" Aug 04 00:09:33.820761 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.820735 2571 scope.go:117] "RemoveContainer" containerID="c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce" Aug 04 00:09:33.821011 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:09:33.820992 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce\": container with ID starting with c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce not found: ID does not exist" containerID="c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce" Aug 04 00:09:33.821087 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.821024 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce"} err="failed to get container status \"c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce\": rpc error: code = NotFound desc = could not find container \"c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce\": container with ID starting with c839616bd607a058f0cbd42bfa1cfb33183f051ae434a6d710d6422b7064bcce not found: ID does not exist" Aug 04 00:09:33.833749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.833722 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848df65887-xxjzk"] Aug 04 00:09:33.836315 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:33.836294 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848df65887-xxjzk"] Aug 04 00:09:34.375796 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:09:34.375764 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a21d8de9-13da-4355-9833-48813465d31e" path="/var/lib/kubelet/pods/a21d8de9-13da-4355-9833-48813465d31e/volumes" Aug 04 00:10:03.859794 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.859758 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7"] Aug 04 00:10:03.860254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.860053 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21d8de9-13da-4355-9833-48813465d31e" containerName="console" Aug 04 00:10:03.860254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.860063 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21d8de9-13da-4355-9833-48813465d31e" containerName="console" Aug 04 00:10:03.860254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.860136 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21d8de9-13da-4355-9833-48813465d31e" containerName="console" Aug 04 00:10:03.862946 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.862930 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:03.865327 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.865304 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 00:10:03.865458 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.865357 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 00:10:03.865993 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.865973 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ln88f\"" Aug 04 00:10:03.870323 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.870301 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7"] Aug 04 00:10:03.976590 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.976515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:03.976590 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.976560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:03.976780 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:03.976619 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4jk\" (UniqueName: \"kubernetes.io/projected/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-kube-api-access-th4jk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.077031 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.076990 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4jk\" (UniqueName: \"kubernetes.io/projected/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-kube-api-access-th4jk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.077239 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.077058 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.077239 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.077082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.077536 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.077516 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.077588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.077520 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.085164 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.085131 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4jk\" (UniqueName: \"kubernetes.io/projected/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-kube-api-access-th4jk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.172628 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.172529 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:04.304414 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.304375 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7"] Aug 04 00:10:04.307497 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:10:04.307471 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e2dfcc_befb_4a3f_b7ab_dbe1ffd2be72.slice/crio-cbae9764b472c812c8a471f1ab850b3a3946dc78357c4061aa736440c06d7793 WatchSource:0}: Error finding container cbae9764b472c812c8a471f1ab850b3a3946dc78357c4061aa736440c06d7793: Status 404 returned error can't find the container with id cbae9764b472c812c8a471f1ab850b3a3946dc78357c4061aa736440c06d7793 Aug 04 00:10:04.309594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.309573 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:10:04.928029 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:04.927993 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" event={"ID":"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72","Type":"ContainerStarted","Data":"cbae9764b472c812c8a471f1ab850b3a3946dc78357c4061aa736440c06d7793"} Aug 04 00:10:11.957983 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:11.957943 2571 generic.go:356] "Generic (PLEG): container finished" podID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerID="4b3827143d0e338f1d9cc7e4e591caed394b421c08ce33451f9046cbd6d20c6c" exitCode=0 Aug 04 00:10:11.958454 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:11.958030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" event={"ID":"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72","Type":"ContainerDied","Data":"4b3827143d0e338f1d9cc7e4e591caed394b421c08ce33451f9046cbd6d20c6c"} Aug 04 00:10:14.972679 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:14.972639 2571 generic.go:356] "Generic (PLEG): container finished" podID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerID="a45060dec300c277f0e320cd1dce9744053c2d0a443bec8404bd978cb92489d4" exitCode=0 Aug 04 00:10:14.973145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:14.972744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" event={"ID":"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72","Type":"ContainerDied","Data":"a45060dec300c277f0e320cd1dce9744053c2d0a443bec8404bd978cb92489d4"} Aug 04 00:10:23.006699 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:23.006655 2571 generic.go:356] "Generic (PLEG): container finished" podID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerID="251545775abae96f20e8fb7592f4370ea8daec4b5e1c0cce7344fb01700deb20" exitCode=0 Aug 04 00:10:23.007077 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:23.006738 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" event={"ID":"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72","Type":"ContainerDied","Data":"251545775abae96f20e8fb7592f4370ea8daec4b5e1c0cce7344fb01700deb20"} Aug 04 00:10:24.132831 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.132804 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:24.156547 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.156520 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-bundle\") pod \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " Aug 04 00:10:24.156666 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.156605 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th4jk\" (UniqueName: \"kubernetes.io/projected/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-kube-api-access-th4jk\") pod \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " Aug 04 00:10:24.156666 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.156646 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-util\") pod \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\" (UID: \"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72\") " Aug 04 00:10:24.157403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.157372 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-bundle" (OuterVolumeSpecName: "bundle") pod "c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" (UID: "c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:10:24.158790 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.158760 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-kube-api-access-th4jk" (OuterVolumeSpecName: "kube-api-access-th4jk") pod "c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" (UID: "c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72"). InnerVolumeSpecName "kube-api-access-th4jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:10:24.160901 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.160881 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-util" (OuterVolumeSpecName: "util") pod "c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" (UID: "c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:10:24.257773 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.257744 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:10:24.257773 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.257765 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:10:24.257773 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:24.257775 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4jk\" (UniqueName: \"kubernetes.io/projected/c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72-kube-api-access-th4jk\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:10:25.016193 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:25.016161 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" Aug 04 00:10:25.016193 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:25.016176 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l9wl7" event={"ID":"c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72","Type":"ContainerDied","Data":"cbae9764b472c812c8a471f1ab850b3a3946dc78357c4061aa736440c06d7793"} Aug 04 00:10:25.016445 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:25.016224 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbae9764b472c812c8a471f1ab850b3a3946dc78357c4061aa736440c06d7793" Aug 04 00:10:31.957642 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.957606 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr"] Aug 04 00:10:31.958022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.957955 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="pull" Aug 04 00:10:31.958022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.957968 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="pull" Aug 04 00:10:31.958022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.957989 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="util" Aug 04 00:10:31.958022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.957994 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="util" Aug 04 00:10:31.958022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.958005 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="extract" Aug 04 00:10:31.958022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.958011 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="extract" Aug 04 00:10:31.958266 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.958056 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e2dfcc-befb-4a3f-b7ab-dbe1ffd2be72" containerName="extract" Aug 04 00:10:31.962024 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.962007 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:31.964308 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.964288 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-llj48\"" Aug 04 00:10:31.964564 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.964547 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 00:10:31.964631 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.964582 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:10:31.973530 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:31.973511 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr"] Aug 04 00:10:32.010159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.010127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3f0f7c8-9924-4409-ac12-b811967e6453-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzbgr\" (UID: \"e3f0f7c8-9924-4409-ac12-b811967e6453\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.010299 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.010212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbfv\" (UniqueName: \"kubernetes.io/projected/e3f0f7c8-9924-4409-ac12-b811967e6453-kube-api-access-pkbfv\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzbgr\" (UID: \"e3f0f7c8-9924-4409-ac12-b811967e6453\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.110737 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.110704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3f0f7c8-9924-4409-ac12-b811967e6453-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzbgr\" (UID: \"e3f0f7c8-9924-4409-ac12-b811967e6453\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.110910 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.110753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbfv\" (UniqueName: \"kubernetes.io/projected/e3f0f7c8-9924-4409-ac12-b811967e6453-kube-api-access-pkbfv\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzbgr\" (UID: \"e3f0f7c8-9924-4409-ac12-b811967e6453\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.111194 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.111175 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3f0f7c8-9924-4409-ac12-b811967e6453-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzbgr\" (UID: \"e3f0f7c8-9924-4409-ac12-b811967e6453\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.119363 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.119335 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbfv\" (UniqueName: \"kubernetes.io/projected/e3f0f7c8-9924-4409-ac12-b811967e6453-kube-api-access-pkbfv\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzbgr\" (UID: \"e3f0f7c8-9924-4409-ac12-b811967e6453\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.270770 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.270742 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" Aug 04 00:10:32.403222 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:32.403191 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr"] Aug 04 00:10:32.406088 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:10:32.406063 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f0f7c8_9924_4409_ac12_b811967e6453.slice/crio-ac609eabe44ee0cc0cd705a61b6b4ec3dc429f37ade9686f69bc8b2d6ba773c0 WatchSource:0}: Error finding container ac609eabe44ee0cc0cd705a61b6b4ec3dc429f37ade9686f69bc8b2d6ba773c0: Status 404 returned error can't find the container with id ac609eabe44ee0cc0cd705a61b6b4ec3dc429f37ade9686f69bc8b2d6ba773c0 Aug 04 00:10:33.050163 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:33.050129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" event={"ID":"e3f0f7c8-9924-4409-ac12-b811967e6453","Type":"ContainerStarted","Data":"ac609eabe44ee0cc0cd705a61b6b4ec3dc429f37ade9686f69bc8b2d6ba773c0"} Aug 04 00:10:35.060267 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.060225 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" event={"ID":"e3f0f7c8-9924-4409-ac12-b811967e6453","Type":"ContainerStarted","Data":"343a63c6266e7e518322d78b17bb620edfd56b44fa735e4e5730c0686ac987bf"} Aug 04 00:10:35.087207 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.087159 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzbgr" podStartSLOduration=2.161240399 podStartE2EDuration="4.087142105s" totalImagesPullingTime="1.925901706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:10:35.086129847 +0000 UTC m=+473.257163182" watchObservedRunningTime="2026-08-04 00:10:35.087142105 +0000 UTC m=+473.258175402" Aug 04 00:10:35.435882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.435802 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j6xmj"] Aug 04 00:10:35.437918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.437900 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.440995 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.440975 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 00:10:35.441137 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.441006 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ttmh8\"" Aug 04 00:10:35.441592 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.441577 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 00:10:35.450641 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.450620 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j6xmj"] Aug 04 00:10:35.533894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.533864 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47zj\" (UniqueName: \"kubernetes.io/projected/253814a6-317b-429b-9d43-2d0f43ac96ae-kube-api-access-z47zj\") pod \"cert-manager-webhook-6486ffd997-j6xmj\" (UID: \"253814a6-317b-429b-9d43-2d0f43ac96ae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.534057 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.533900 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/253814a6-317b-429b-9d43-2d0f43ac96ae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j6xmj\" (UID: \"253814a6-317b-429b-9d43-2d0f43ac96ae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.634927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.634893 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47zj\" (UniqueName: \"kubernetes.io/projected/253814a6-317b-429b-9d43-2d0f43ac96ae-kube-api-access-z47zj\") pod \"cert-manager-webhook-6486ffd997-j6xmj\" (UID: \"253814a6-317b-429b-9d43-2d0f43ac96ae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.634927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.634929 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/253814a6-317b-429b-9d43-2d0f43ac96ae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j6xmj\" (UID: \"253814a6-317b-429b-9d43-2d0f43ac96ae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.646895 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.646861 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47zj\" (UniqueName: \"kubernetes.io/projected/253814a6-317b-429b-9d43-2d0f43ac96ae-kube-api-access-z47zj\") pod \"cert-manager-webhook-6486ffd997-j6xmj\" (UID: \"253814a6-317b-429b-9d43-2d0f43ac96ae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.647967 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.647948 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/253814a6-317b-429b-9d43-2d0f43ac96ae-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j6xmj\" (UID: \"253814a6-317b-429b-9d43-2d0f43ac96ae\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.746510 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.746431 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:35.870016 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:35.869988 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j6xmj"] Aug 04 00:10:35.873037 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:10:35.873009 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod253814a6_317b_429b_9d43_2d0f43ac96ae.slice/crio-6003d29142205db35dcc73c7843d24cdca97c4832f98ea482a277bb0fc1cd033 WatchSource:0}: Error finding container 6003d29142205db35dcc73c7843d24cdca97c4832f98ea482a277bb0fc1cd033: Status 404 returned error can't find the container with id 6003d29142205db35dcc73c7843d24cdca97c4832f98ea482a277bb0fc1cd033 Aug 04 00:10:36.065811 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.065774 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" event={"ID":"253814a6-317b-429b-9d43-2d0f43ac96ae","Type":"ContainerStarted","Data":"6003d29142205db35dcc73c7843d24cdca97c4832f98ea482a277bb0fc1cd033"} Aug 04 00:10:36.205398 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.205363 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd"] Aug 04 00:10:36.208507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.208491 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.211038 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.211017 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 00:10:36.211196 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.211075 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 00:10:36.211255 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.211195 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ln88f\"" Aug 04 00:10:36.219124 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.219086 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd"] Aug 04 00:10:36.239265 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.239240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4v22\" (UniqueName: \"kubernetes.io/projected/85b97c1d-625c-4095-b315-3dcfbc01e803-kube-api-access-t4v22\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.239379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.239270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.239379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.239343 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.341497 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.340606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.341497 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.340710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v22\" (UniqueName: \"kubernetes.io/projected/85b97c1d-625c-4095-b315-3dcfbc01e803-kube-api-access-t4v22\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.341497 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.340748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.341497 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.341259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.345182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.342412 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.353815 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.353794 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4v22\" (UniqueName: \"kubernetes.io/projected/85b97c1d-625c-4095-b315-3dcfbc01e803-kube-api-access-t4v22\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.518358 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.518327 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:36.643060 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:36.643029 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd"] Aug 04 00:10:36.646854 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:10:36.646823 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b97c1d_625c_4095_b315_3dcfbc01e803.slice/crio-f81af8b4298e656be6716984bb0543aa2196f4485a02a2921857a48390aa0d2d WatchSource:0}: Error finding container f81af8b4298e656be6716984bb0543aa2196f4485a02a2921857a48390aa0d2d: Status 404 returned error can't find the container with id f81af8b4298e656be6716984bb0543aa2196f4485a02a2921857a48390aa0d2d Aug 04 00:10:37.072039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:37.071995 2571 generic.go:356] "Generic (PLEG): container finished" podID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerID="b63fafbb28758d946d10c25cbe78e23eff8e7b7ddbf0aacaa6db2086ed1b22f2" exitCode=0 Aug 04 00:10:37.072039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:37.072043 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" event={"ID":"85b97c1d-625c-4095-b315-3dcfbc01e803","Type":"ContainerDied","Data":"b63fafbb28758d946d10c25cbe78e23eff8e7b7ddbf0aacaa6db2086ed1b22f2"} Aug 04 00:10:37.072568 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:37.072090 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" event={"ID":"85b97c1d-625c-4095-b315-3dcfbc01e803","Type":"ContainerStarted","Data":"f81af8b4298e656be6716984bb0543aa2196f4485a02a2921857a48390aa0d2d"} Aug 04 00:10:38.114357 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.114322 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7"] Aug 04 00:10:38.116968 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.116946 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.119117 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.119082 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cb9cq\"" Aug 04 00:10:38.127720 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.127695 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7"] Aug 04 00:10:38.157937 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.157906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71749c1d-e095-49df-8484-557a27bd2936-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5k5m7\" (UID: \"71749c1d-e095-49df-8484-557a27bd2936\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.158039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.157945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnms\" (UniqueName: \"kubernetes.io/projected/71749c1d-e095-49df-8484-557a27bd2936-kube-api-access-kjnms\") pod \"cert-manager-cainjector-559cdd8c87-5k5m7\" (UID: \"71749c1d-e095-49df-8484-557a27bd2936\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.258679 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.258643 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71749c1d-e095-49df-8484-557a27bd2936-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5k5m7\" (UID: \"71749c1d-e095-49df-8484-557a27bd2936\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.258679 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.258682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnms\" (UniqueName: \"kubernetes.io/projected/71749c1d-e095-49df-8484-557a27bd2936-kube-api-access-kjnms\") pod \"cert-manager-cainjector-559cdd8c87-5k5m7\" (UID: \"71749c1d-e095-49df-8484-557a27bd2936\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.267157 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.267129 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71749c1d-e095-49df-8484-557a27bd2936-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5k5m7\" (UID: \"71749c1d-e095-49df-8484-557a27bd2936\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.267343 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.267320 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnms\" (UniqueName: \"kubernetes.io/projected/71749c1d-e095-49df-8484-557a27bd2936-kube-api-access-kjnms\") pod \"cert-manager-cainjector-559cdd8c87-5k5m7\" (UID: \"71749c1d-e095-49df-8484-557a27bd2936\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.427690 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.427614 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" Aug 04 00:10:38.696835 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:38.696811 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7"] Aug 04 00:10:38.699626 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:10:38.699604 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71749c1d_e095_49df_8484_557a27bd2936.slice/crio-f95c5f503037cbdec671b1f7907a51095402baa14830be752fb8ee1483ab8189 WatchSource:0}: Error finding container f95c5f503037cbdec671b1f7907a51095402baa14830be752fb8ee1483ab8189: Status 404 returned error can't find the container with id f95c5f503037cbdec671b1f7907a51095402baa14830be752fb8ee1483ab8189 Aug 04 00:10:39.082638 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:39.082596 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" event={"ID":"71749c1d-e095-49df-8484-557a27bd2936","Type":"ContainerStarted","Data":"4a3d3d0039c86aa7e621e3d99afba2e2d2820abf3ed36237631785f7009a9982"} Aug 04 00:10:39.082847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:39.082651 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" event={"ID":"71749c1d-e095-49df-8484-557a27bd2936","Type":"ContainerStarted","Data":"f95c5f503037cbdec671b1f7907a51095402baa14830be752fb8ee1483ab8189"} Aug 04 00:10:39.084441 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:39.084407 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" event={"ID":"253814a6-317b-429b-9d43-2d0f43ac96ae","Type":"ContainerStarted","Data":"3fd88fba74d9ec8549ac7a8e39d5eb18133e271e3039bf25ba5bebd29dbff72b"} Aug 04 00:10:39.084648 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:39.084624 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:10:39.099548 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:39.099502 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5k5m7" podStartSLOduration=1.099488676 podStartE2EDuration="1.099488676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:10:39.097445301 +0000 UTC m=+477.268478605" watchObservedRunningTime="2026-08-04 00:10:39.099488676 +0000 UTC m=+477.270521983" Aug 04 00:10:39.118828 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:39.118777 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" podStartSLOduration=1.381812471 podStartE2EDuration="4.118759701s" totalImagesPullingTime="2.73694723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:10:39.118304253 +0000 UTC m=+477.289337559" watchObservedRunningTime="2026-08-04 00:10:39.118759701 +0000 UTC m=+477.289793010" Aug 04 00:10:41.094790 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:41.094749 2571 generic.go:356] "Generic (PLEG): container finished" podID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerID="7dd07e1ff27c23de2687d567e8ff99c3b39be36cf41b9da56d5a29fc0fafe722" exitCode=0 Aug 04 00:10:41.095176 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:41.094832 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" event={"ID":"85b97c1d-625c-4095-b315-3dcfbc01e803","Type":"ContainerDied","Data":"7dd07e1ff27c23de2687d567e8ff99c3b39be36cf41b9da56d5a29fc0fafe722"} Aug 04 00:10:42.101636 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:42.101595 2571 generic.go:356] "Generic (PLEG): container finished" podID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerID="d3f7c80093de9ab66559ea0371575f358cd439aa1ebde49afd6cb179793304e5" exitCode=0 Aug 04 00:10:42.102025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:42.101646 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" event={"ID":"85b97c1d-625c-4095-b315-3dcfbc01e803","Type":"ContainerDied","Data":"d3f7c80093de9ab66559ea0371575f358cd439aa1ebde49afd6cb179793304e5"} Aug 04 00:10:43.236095 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.236071 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:43.301340 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.301311 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4v22\" (UniqueName: \"kubernetes.io/projected/85b97c1d-625c-4095-b315-3dcfbc01e803-kube-api-access-t4v22\") pod \"85b97c1d-625c-4095-b315-3dcfbc01e803\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " Aug 04 00:10:43.301506 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.301400 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-bundle\") pod \"85b97c1d-625c-4095-b315-3dcfbc01e803\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " Aug 04 00:10:43.301506 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.301445 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-util\") pod \"85b97c1d-625c-4095-b315-3dcfbc01e803\" (UID: \"85b97c1d-625c-4095-b315-3dcfbc01e803\") " Aug 04 00:10:43.301807 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.301779 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-bundle" (OuterVolumeSpecName: "bundle") pod "85b97c1d-625c-4095-b315-3dcfbc01e803" (UID: "85b97c1d-625c-4095-b315-3dcfbc01e803"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:10:43.303466 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.303442 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b97c1d-625c-4095-b315-3dcfbc01e803-kube-api-access-t4v22" (OuterVolumeSpecName: "kube-api-access-t4v22") pod "85b97c1d-625c-4095-b315-3dcfbc01e803" (UID: "85b97c1d-625c-4095-b315-3dcfbc01e803"). InnerVolumeSpecName "kube-api-access-t4v22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:10:43.308796 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.308770 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-util" (OuterVolumeSpecName: "util") pod "85b97c1d-625c-4095-b315-3dcfbc01e803" (UID: "85b97c1d-625c-4095-b315-3dcfbc01e803"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:10:43.402383 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.402312 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:10:43.402383 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.402337 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/85b97c1d-625c-4095-b315-3dcfbc01e803-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:10:43.402383 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:43.402346 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4v22\" (UniqueName: \"kubernetes.io/projected/85b97c1d-625c-4095-b315-3dcfbc01e803-kube-api-access-t4v22\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:10:44.112316 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:44.112278 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" event={"ID":"85b97c1d-625c-4095-b315-3dcfbc01e803","Type":"ContainerDied","Data":"f81af8b4298e656be6716984bb0543aa2196f4485a02a2921857a48390aa0d2d"} Aug 04 00:10:44.112316 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:44.112306 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftjhpd" Aug 04 00:10:44.112316 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:44.112326 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81af8b4298e656be6716984bb0543aa2196f4485a02a2921857a48390aa0d2d" Aug 04 00:10:45.091391 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:10:45.091362 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j6xmj" Aug 04 00:11:04.845800 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.845761 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv"] Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846236 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="util" Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846256 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="util" Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846270 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="extract" Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846277 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="extract" Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846307 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="pull" Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846316 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="pull" Aug 04 00:11:04.846425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.846389 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b97c1d-625c-4095-b315-3dcfbc01e803" containerName="extract" Aug 04 00:11:04.850842 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.850820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:04.853388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.853370 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 00:11:04.854006 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.853989 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ln88f\"" Aug 04 00:11:04.854090 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.854011 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 00:11:04.856985 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.856950 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv"] Aug 04 00:11:04.970238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.970203 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twspq\" (UniqueName: \"kubernetes.io/projected/8367a43f-317c-49e0-b3b8-28b289a0410b-kube-api-access-twspq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:04.970379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.970244 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:04.970379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:04.970262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.071064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.071020 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twspq\" (UniqueName: \"kubernetes.io/projected/8367a43f-317c-49e0-b3b8-28b289a0410b-kube-api-access-twspq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.071064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.071070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.071240 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.071089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.071414 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.071400 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.071522 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.071504 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.079937 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.079914 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twspq\" (UniqueName: \"kubernetes.io/projected/8367a43f-317c-49e0-b3b8-28b289a0410b-kube-api-access-twspq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.160335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.160262 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:05.288873 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:05.288840 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv"] Aug 04 00:11:05.292759 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:11:05.292732 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8367a43f_317c_49e0_b3b8_28b289a0410b.slice/crio-cd5bed400eb8b6056f282f01bef9dbd4eb77422caba9a5068b85908b438ecc87 WatchSource:0}: Error finding container cd5bed400eb8b6056f282f01bef9dbd4eb77422caba9a5068b85908b438ecc87: Status 404 returned error can't find the container with id cd5bed400eb8b6056f282f01bef9dbd4eb77422caba9a5068b85908b438ecc87 Aug 04 00:11:06.159210 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.159177 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h"] Aug 04 00:11:06.161488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.161472 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.163829 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.163807 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 00:11:06.164658 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.164625 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 00:11:06.164754 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.164651 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 00:11:06.164754 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.164705 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tp5bz\"" Aug 04 00:11:06.164754 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.164713 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 00:11:06.164899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.164784 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 00:11:06.172453 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.172433 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h"] Aug 04 00:11:06.203633 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.203595 2571 generic.go:356] "Generic (PLEG): container finished" podID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerID="a1318e9909e0665779db58c5229d1da11e331c7c5cd2e0f191a849bc5cb5b546" exitCode=0 Aug 04 00:11:06.203767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.203654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" event={"ID":"8367a43f-317c-49e0-b3b8-28b289a0410b","Type":"ContainerDied","Data":"a1318e9909e0665779db58c5229d1da11e331c7c5cd2e0f191a849bc5cb5b546"} Aug 04 00:11:06.203767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.203710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" event={"ID":"8367a43f-317c-49e0-b3b8-28b289a0410b","Type":"ContainerStarted","Data":"cd5bed400eb8b6056f282f01bef9dbd4eb77422caba9a5068b85908b438ecc87"} Aug 04 00:11:06.280812 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.280786 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746dm\" (UniqueName: \"kubernetes.io/projected/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-kube-api-access-746dm\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.280966 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.280825 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-cert\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.280966 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.280855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-manager-config\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.281055 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.280971 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-metrics-cert\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.381791 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.381712 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746dm\" (UniqueName: \"kubernetes.io/projected/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-kube-api-access-746dm\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.381791 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.381756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-cert\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.381993 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.381826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-manager-config\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.381993 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.381867 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-metrics-cert\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.382484 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.382461 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-manager-config\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.384362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.384343 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-cert\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.384415 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.384352 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-metrics-cert\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.389746 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.389724 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-746dm\" (UniqueName: \"kubernetes.io/projected/fe785b11-fa5f-453d-ada5-29e8ce2dca2a-kube-api-access-746dm\") pod \"lws-controller-manager-7cfb669bf4-5kw7h\" (UID: \"fe785b11-fa5f-453d-ada5-29e8ce2dca2a\") " pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.472091 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.472024 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:06.599448 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:06.599416 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h"] Aug 04 00:11:06.602942 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:11:06.602915 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe785b11_fa5f_453d_ada5_29e8ce2dca2a.slice/crio-0e5de0fc29523e34f4d489b8163ee98e8db0793cd53474c0e4b1928be3f182e4 WatchSource:0}: Error finding container 0e5de0fc29523e34f4d489b8163ee98e8db0793cd53474c0e4b1928be3f182e4: Status 404 returned error can't find the container with id 0e5de0fc29523e34f4d489b8163ee98e8db0793cd53474c0e4b1928be3f182e4 Aug 04 00:11:07.209262 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:07.209167 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" event={"ID":"fe785b11-fa5f-453d-ada5-29e8ce2dca2a","Type":"ContainerStarted","Data":"0e5de0fc29523e34f4d489b8163ee98e8db0793cd53474c0e4b1928be3f182e4"} Aug 04 00:11:07.210997 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:07.210968 2571 generic.go:356] "Generic (PLEG): container finished" podID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerID="19f29216918b092b6dc548a2c6ad1191b233515963a1fe6991ca8cefa3f171a0" exitCode=0 Aug 04 00:11:07.211143 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:07.211045 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" event={"ID":"8367a43f-317c-49e0-b3b8-28b289a0410b","Type":"ContainerDied","Data":"19f29216918b092b6dc548a2c6ad1191b233515963a1fe6991ca8cefa3f171a0"} Aug 04 00:11:08.217411 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:08.217374 2571 generic.go:356] "Generic (PLEG): container finished" podID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerID="6ca73da30ba2b648fbc87c7deed8b9aa0c4da04fbee5292d2ad6df22e9c21d75" exitCode=0 Aug 04 00:11:08.217800 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:08.217437 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" event={"ID":"8367a43f-317c-49e0-b3b8-28b289a0410b","Type":"ContainerDied","Data":"6ca73da30ba2b648fbc87c7deed8b9aa0c4da04fbee5292d2ad6df22e9c21d75"} Aug 04 00:11:09.223457 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.223407 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" event={"ID":"fe785b11-fa5f-453d-ada5-29e8ce2dca2a","Type":"ContainerStarted","Data":"f46de426b7c1d28719144d61537186ff54e4f06e7d4fd4c5a1bfe688111a7a70"} Aug 04 00:11:09.223879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.223648 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:09.241853 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.241788 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" podStartSLOduration=1.6149808239999999 podStartE2EDuration="3.241768753s" totalImagesPullingTime="1.626787929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:09.241513495 +0000 UTC m=+507.412546838" watchObservedRunningTime="2026-08-04 00:11:09.241768753 +0000 UTC m=+507.412802060" Aug 04 00:11:09.352452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.352427 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:09.516871 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.516783 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twspq\" (UniqueName: \"kubernetes.io/projected/8367a43f-317c-49e0-b3b8-28b289a0410b-kube-api-access-twspq\") pod \"8367a43f-317c-49e0-b3b8-28b289a0410b\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " Aug 04 00:11:09.516871 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.516842 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-util\") pod \"8367a43f-317c-49e0-b3b8-28b289a0410b\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " Aug 04 00:11:09.517048 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.516960 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-bundle\") pod \"8367a43f-317c-49e0-b3b8-28b289a0410b\" (UID: \"8367a43f-317c-49e0-b3b8-28b289a0410b\") " Aug 04 00:11:09.517696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.517654 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-bundle" (OuterVolumeSpecName: "bundle") pod "8367a43f-317c-49e0-b3b8-28b289a0410b" (UID: "8367a43f-317c-49e0-b3b8-28b289a0410b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:11:09.518991 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.518970 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8367a43f-317c-49e0-b3b8-28b289a0410b-kube-api-access-twspq" (OuterVolumeSpecName: "kube-api-access-twspq") pod "8367a43f-317c-49e0-b3b8-28b289a0410b" (UID: "8367a43f-317c-49e0-b3b8-28b289a0410b"). InnerVolumeSpecName "kube-api-access-twspq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:11:09.522779 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.522750 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-util" (OuterVolumeSpecName: "util") pod "8367a43f-317c-49e0-b3b8-28b289a0410b" (UID: "8367a43f-317c-49e0-b3b8-28b289a0410b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:11:09.618076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.618045 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:11:09.618076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.618071 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twspq\" (UniqueName: \"kubernetes.io/projected/8367a43f-317c-49e0-b3b8-28b289a0410b-kube-api-access-twspq\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:11:09.618076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:09.618081 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8367a43f-317c-49e0-b3b8-28b289a0410b-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:11:10.228971 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:10.228941 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" Aug 04 00:11:10.229392 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:10.228962 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5nrv" event={"ID":"8367a43f-317c-49e0-b3b8-28b289a0410b","Type":"ContainerDied","Data":"cd5bed400eb8b6056f282f01bef9dbd4eb77422caba9a5068b85908b438ecc87"} Aug 04 00:11:10.229392 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:10.229017 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5bed400eb8b6056f282f01bef9dbd4eb77422caba9a5068b85908b438ecc87" Aug 04 00:11:17.555329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555294 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n"] Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555663 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="extract" Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555676 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="extract" Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555687 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="util" Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555697 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="util" Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555707 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="pull" Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555713 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="pull" Aug 04 00:11:17.555840 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.555776 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8367a43f-317c-49e0-b3b8-28b289a0410b" containerName="extract" Aug 04 00:11:17.558457 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.558440 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.563732 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.563706 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 00:11:17.564097 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.564071 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 00:11:17.564226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.564187 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 00:11:17.564226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.564211 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h29qs\"" Aug 04 00:11:17.564623 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.564604 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 00:11:17.577726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.577696 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n"] Aug 04 00:11:17.690019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.689981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.690251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.690033 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.690251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.690136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79c28\" (UniqueName: \"kubernetes.io/projected/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-kube-api-access-79c28\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.690251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.690201 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.690369 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.690297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.791231 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.791196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.791400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.791244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c28\" (UniqueName: \"kubernetes.io/projected/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-kube-api-access-79c28\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.791400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.791287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.791400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.791354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.791400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.791382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.791688 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.791667 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-manifests\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.793727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.793695 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-tmp\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.793939 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.793916 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.793993 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.793979 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.804411 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.804387 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79c28\" (UniqueName: \"kubernetes.io/projected/dc2050fa-6d9b-4e19-b71a-5fdae40301e4-kube-api-access-79c28\") pod \"opendatahub-operator-controller-manager-67b98d597c-xn65n\" (UID: \"dc2050fa-6d9b-4e19-b71a-5fdae40301e4\") " pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:17.869122 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:17.869031 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:18.003539 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:18.003505 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n"] Aug 04 00:11:18.006246 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:11:18.006217 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2050fa_6d9b_4e19_b71a_5fdae40301e4.slice/crio-457ce9b9234860949a87eebe494494e5b6ae8d9abe4b2341c18ee25fe6bebacc WatchSource:0}: Error finding container 457ce9b9234860949a87eebe494494e5b6ae8d9abe4b2341c18ee25fe6bebacc: Status 404 returned error can't find the container with id 457ce9b9234860949a87eebe494494e5b6ae8d9abe4b2341c18ee25fe6bebacc Aug 04 00:11:18.264160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:18.264048 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" event={"ID":"dc2050fa-6d9b-4e19-b71a-5fdae40301e4","Type":"ContainerStarted","Data":"457ce9b9234860949a87eebe494494e5b6ae8d9abe4b2341c18ee25fe6bebacc"} Aug 04 00:11:20.232245 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:20.232212 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cfb669bf4-5kw7h" Aug 04 00:11:21.281417 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:21.281376 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" event={"ID":"dc2050fa-6d9b-4e19-b71a-5fdae40301e4","Type":"ContainerStarted","Data":"d1732a3bf11135e1a09e7a64768ce36658a468bcce96da471fbb6576ccafcd4a"} Aug 04 00:11:22.287982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:22.287940 2571 generic.go:356] "Generic (PLEG): container finished" podID="dc2050fa-6d9b-4e19-b71a-5fdae40301e4" containerID="d1732a3bf11135e1a09e7a64768ce36658a468bcce96da471fbb6576ccafcd4a" exitCode=0 Aug 04 00:11:22.288373 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:22.288030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" event={"ID":"dc2050fa-6d9b-4e19-b71a-5fdae40301e4","Type":"ContainerDied","Data":"d1732a3bf11135e1a09e7a64768ce36658a468bcce96da471fbb6576ccafcd4a"} Aug 04 00:11:23.293961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:23.293918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" event={"ID":"dc2050fa-6d9b-4e19-b71a-5fdae40301e4","Type":"ContainerStarted","Data":"3aee9b0be18058f87e894591a77b43358a08dd16a29dcca4a2e288d6f1a2fc69"} Aug 04 00:11:23.294368 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:23.293985 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:23.315428 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:23.315376 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" podStartSLOduration=2.141612985 podStartE2EDuration="6.315354511s" totalImagesPullingTime="3.12013813s" totalInitContainerRuntime="1.053603396s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:23.313468979 +0000 UTC m=+521.484502285" watchObservedRunningTime="2026-08-04 00:11:23.315354511 +0000 UTC m=+521.486387819" Aug 04 00:11:34.301577 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:34.301496 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67b98d597c-xn65n" Aug 04 00:11:44.501742 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.501702 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5"] Aug 04 00:11:44.510056 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.510030 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.512324 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.512284 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 00:11:44.512434 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.512392 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 00:11:44.512540 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.512526 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 00:11:44.512667 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.512648 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mfbfj\"" Aug 04 00:11:44.512710 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.512696 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 00:11:44.518305 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.518286 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5"] Aug 04 00:11:44.601669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.601669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53eb014e-72e1-47af-968e-602fc81ab637-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.601669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601642 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wk4\" (UniqueName: \"kubernetes.io/projected/53eb014e-72e1-47af-968e-602fc81ab637-kube-api-access-q5wk4\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.601669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53eb014e-72e1-47af-968e-602fc81ab637-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.601961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53eb014e-72e1-47af-968e-602fc81ab637-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.601961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.601961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.601869 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.702970 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.702939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53eb014e-72e1-47af-968e-602fc81ab637-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.702976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.703003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.703028 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.703051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53eb014e-72e1-47af-968e-602fc81ab637-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.703070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wk4\" (UniqueName: \"kubernetes.io/projected/53eb014e-72e1-47af-968e-602fc81ab637-kube-api-access-q5wk4\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.703089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53eb014e-72e1-47af-968e-602fc81ab637-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.703833 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.703809 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53eb014e-72e1-47af-968e-602fc81ab637-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.705435 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.705414 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53eb014e-72e1-47af-968e-602fc81ab637-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.705532 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.705511 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.705626 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.705604 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.705660 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.705622 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53eb014e-72e1-47af-968e-602fc81ab637-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.710951 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.710924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wk4\" (UniqueName: \"kubernetes.io/projected/53eb014e-72e1-47af-968e-602fc81ab637-kube-api-access-q5wk4\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.711232 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.711210 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53eb014e-72e1-47af-968e-602fc81ab637-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jhnq5\" (UID: \"53eb014e-72e1-47af-968e-602fc81ab637\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.820050 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.820013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:44.951979 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:44.951929 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5"] Aug 04 00:11:44.954205 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:11:44.954171 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53eb014e_72e1_47af_968e_602fc81ab637.slice/crio-a5988e2516b5e35e660d46b6860d50aee0dccd911ace94be36ccd7267ebcbe8a WatchSource:0}: Error finding container a5988e2516b5e35e660d46b6860d50aee0dccd911ace94be36ccd7267ebcbe8a: Status 404 returned error can't find the container with id a5988e2516b5e35e660d46b6860d50aee0dccd911ace94be36ccd7267ebcbe8a Aug 04 00:11:45.393062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:45.393024 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" event={"ID":"53eb014e-72e1-47af-968e-602fc81ab637","Type":"ContainerStarted","Data":"a5988e2516b5e35e660d46b6860d50aee0dccd911ace94be36ccd7267ebcbe8a"} Aug 04 00:11:47.637906 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:47.637863 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 00:11:47.638213 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:47.637937 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 04 00:11:48.409046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:48.409005 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" event={"ID":"53eb014e-72e1-47af-968e-602fc81ab637","Type":"ContainerStarted","Data":"4e0d813c3480e3178c0724a93e404638b8a3afee5a09dda456da9a92315b23c0"} Aug 04 00:11:48.409267 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:48.409175 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:48.410894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:48.410874 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" Aug 04 00:11:48.435490 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:11:48.435435 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jhnq5" podStartSLOduration=1.754151952 podStartE2EDuration="4.43541859s" totalImagesPullingTime="2.681266638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:11:48.434335765 +0000 UTC m=+546.605369064" watchObservedRunningTime="2026-08-04 00:11:48.43541859 +0000 UTC m=+546.606451896" Aug 04 00:12:14.229712 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.229672 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-558f4699d9-lgczg"] Aug 04 00:12:14.233001 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.232979 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.245166 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.245141 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-558f4699d9-lgczg"] Aug 04 00:12:14.356400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885m8\" (UniqueName: \"kubernetes.io/projected/992df4df-7120-4af9-bd77-043471e6ba41-kube-api-access-885m8\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.356400 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356402 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-trusted-ca-bundle\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.356657 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356445 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/992df4df-7120-4af9-bd77-043471e6ba41-console-serving-cert\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.356657 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-service-ca\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.356657 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356509 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-oauth-serving-cert\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.356657 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-console-config\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.356657 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.356557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/992df4df-7120-4af9-bd77-043471e6ba41-console-oauth-config\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.457937 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.457895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885m8\" (UniqueName: \"kubernetes.io/projected/992df4df-7120-4af9-bd77-043471e6ba41-kube-api-access-885m8\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.457986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-trusted-ca-bundle\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458036 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/992df4df-7120-4af9-bd77-043471e6ba41-console-serving-cert\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-service-ca\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458126 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-oauth-serving-cert\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458146 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-console-config\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458458 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458167 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/992df4df-7120-4af9-bd77-043471e6ba41-console-oauth-config\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.458963 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458930 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-service-ca\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.459078 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.458981 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-console-config\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.459078 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.459044 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-oauth-serving-cert\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.459200 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.459052 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992df4df-7120-4af9-bd77-043471e6ba41-trusted-ca-bundle\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.461037 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.461019 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/992df4df-7120-4af9-bd77-043471e6ba41-console-oauth-config\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.461292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.461269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/992df4df-7120-4af9-bd77-043471e6ba41-console-serving-cert\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.465899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.465877 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-885m8\" (UniqueName: \"kubernetes.io/projected/992df4df-7120-4af9-bd77-043471e6ba41-kube-api-access-885m8\") pod \"console-558f4699d9-lgczg\" (UID: \"992df4df-7120-4af9-bd77-043471e6ba41\") " pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.543053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.543032 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:14.674886 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:14.674860 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-558f4699d9-lgczg"] Aug 04 00:12:14.677160 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:12:14.677131 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992df4df_7120_4af9_bd77_043471e6ba41.slice/crio-22e1d7db6ca433108dc34a84827bd702028d46594003b010827a68773f339d8a WatchSource:0}: Error finding container 22e1d7db6ca433108dc34a84827bd702028d46594003b010827a68773f339d8a: Status 404 returned error can't find the container with id 22e1d7db6ca433108dc34a84827bd702028d46594003b010827a68773f339d8a Aug 04 00:12:15.530518 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:15.530484 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-558f4699d9-lgczg" event={"ID":"992df4df-7120-4af9-bd77-043471e6ba41","Type":"ContainerStarted","Data":"e4990b4ae8ca5784166b7e5f14475bd69e6877493bf458609eb16c4256174cd1"} Aug 04 00:12:15.530916 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:15.530533 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-558f4699d9-lgczg" event={"ID":"992df4df-7120-4af9-bd77-043471e6ba41","Type":"ContainerStarted","Data":"22e1d7db6ca433108dc34a84827bd702028d46594003b010827a68773f339d8a"} Aug 04 00:12:24.543748 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:24.543709 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:24.544256 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:24.543785 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:24.548418 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:24.548396 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:24.567786 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:24.567740 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-558f4699d9-lgczg" podStartSLOduration=10.567726823 podStartE2EDuration="10.567726823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:12:15.550699862 +0000 UTC m=+573.721733169" watchObservedRunningTime="2026-08-04 00:12:24.567726823 +0000 UTC m=+582.738760130" Aug 04 00:12:24.575879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:24.575855 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-558f4699d9-lgczg" Aug 04 00:12:24.621072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:24.621040 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57cb95b47-jxk2w"] Aug 04 00:12:40.861822 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.861787 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr"] Aug 04 00:12:40.865329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.865307 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:40.867404 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.867384 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 00:12:40.867937 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.867922 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ln88f\"" Aug 04 00:12:40.867937 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.867929 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 00:12:40.873270 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.872902 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr"] Aug 04 00:12:40.958390 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.958354 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n"] Aug 04 00:12:40.961891 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.961875 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:40.969514 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.969488 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n"] Aug 04 00:12:40.982956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.982932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:40.983076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.982976 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvcf\" (UniqueName: \"kubernetes.io/projected/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-kube-api-access-9jvcf\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:40.983076 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:40.982996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.057081 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.057053 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h"] Aug 04 00:12:41.060466 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.060451 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.068378 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.068355 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h"] Aug 04 00:12:41.084269 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084243 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mt45\" (UniqueName: \"kubernetes.io/projected/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-kube-api-access-4mt45\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.084393 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.084393 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084328 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.084393 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvcf\" (UniqueName: \"kubernetes.io/projected/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-kube-api-access-9jvcf\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.084550 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.084550 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084481 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.084691 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084673 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.084727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.084674 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.092278 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.092260 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvcf\" (UniqueName: \"kubernetes.io/projected/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-kube-api-access-9jvcf\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.156450 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.156389 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb"] Aug 04 00:12:41.159918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.159902 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.169272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.169247 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb"] Aug 04 00:12:41.176441 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.176418 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:41.185850 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.185830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mt45\" (UniqueName: \"kubernetes.io/projected/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-kube-api-access-4mt45\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.185949 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.185871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.185949 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.185916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.186057 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.185947 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4jh\" (UniqueName: \"kubernetes.io/projected/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-kube-api-access-wf4jh\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.186057 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.185987 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.186178 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.186068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.186383 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.186362 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.186427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.186408 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.194360 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.194334 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mt45\" (UniqueName: \"kubernetes.io/projected/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-kube-api-access-4mt45\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.271509 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.271467 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:41.287320 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.287430 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287367 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqzs\" (UniqueName: \"kubernetes.io/projected/17988a8e-99f5-4096-be7d-7fee2a0cb22f-kube-api-access-7vqzs\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.287430 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.287545 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287449 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.287584 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.287618 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287604 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4jh\" (UniqueName: \"kubernetes.io/projected/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-kube-api-access-wf4jh\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.287742 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.287716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.288260 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.288238 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.295621 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.295596 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4jh\" (UniqueName: \"kubernetes.io/projected/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-kube-api-access-wf4jh\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.312344 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.312268 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr"] Aug 04 00:12:41.314480 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:12:41.314456 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d09f7f5_d7bc_4e17_9738_278eb8564ae1.slice/crio-e534603f954d472305bc5607dc7eee0762af04363b3d0f2769d2f5c5bbc16bbb WatchSource:0}: Error finding container e534603f954d472305bc5607dc7eee0762af04363b3d0f2769d2f5c5bbc16bbb: Status 404 returned error can't find the container with id e534603f954d472305bc5607dc7eee0762af04363b3d0f2769d2f5c5bbc16bbb Aug 04 00:12:41.370588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.370563 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:41.389049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.388671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqzs\" (UniqueName: \"kubernetes.io/projected/17988a8e-99f5-4096-be7d-7fee2a0cb22f-kube-api-access-7vqzs\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.389049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.388749 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.389049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.388785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.389295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.389216 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.389295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.389255 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.397311 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.397287 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqzs\" (UniqueName: \"kubernetes.io/projected/17988a8e-99f5-4096-be7d-7fee2a0cb22f-kube-api-access-7vqzs\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.412657 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.412631 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n"] Aug 04 00:12:41.416441 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:12:41.416377 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae66ba65_c45e_4c4f_a17c_2a09cf21a70a.slice/crio-85b79daf2c1fdb066a9a9ea6ff56f9047c57f8495ec5e2a8ebd5fef9c9868cc1 WatchSource:0}: Error finding container 85b79daf2c1fdb066a9a9ea6ff56f9047c57f8495ec5e2a8ebd5fef9c9868cc1: Status 404 returned error can't find the container with id 85b79daf2c1fdb066a9a9ea6ff56f9047c57f8495ec5e2a8ebd5fef9c9868cc1 Aug 04 00:12:41.469706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.469668 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:41.525728 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.525696 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h"] Aug 04 00:12:41.540901 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:12:41.540874 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5a3023_1391_4b03_9f18_2fa3ad4a8ddf.slice/crio-d71e28b5f9c0ee29f3653ce9e63b603604deabae5f72755b089445694258bf7f WatchSource:0}: Error finding container d71e28b5f9c0ee29f3653ce9e63b603604deabae5f72755b089445694258bf7f: Status 404 returned error can't find the container with id d71e28b5f9c0ee29f3653ce9e63b603604deabae5f72755b089445694258bf7f Aug 04 00:12:41.617712 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.617685 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb"] Aug 04 00:12:41.621372 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:12:41.621336 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17988a8e_99f5_4096_be7d_7fee2a0cb22f.slice/crio-04eb34f80780cf0341548d1212889380d3f91ccf381f100093151ea1db796e7c WatchSource:0}: Error finding container 04eb34f80780cf0341548d1212889380d3f91ccf381f100093151ea1db796e7c: Status 404 returned error can't find the container with id 04eb34f80780cf0341548d1212889380d3f91ccf381f100093151ea1db796e7c Aug 04 00:12:41.653394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.653351 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" event={"ID":"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf","Type":"ContainerStarted","Data":"0487076c04b1c5f736a4ca1bad4535c1563c1b3f33b0fa2db5e9a54dd077ef9e"} Aug 04 00:12:41.653513 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.653428 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" event={"ID":"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf","Type":"ContainerStarted","Data":"d71e28b5f9c0ee29f3653ce9e63b603604deabae5f72755b089445694258bf7f"} Aug 04 00:12:41.654871 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.654838 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" event={"ID":"17988a8e-99f5-4096-be7d-7fee2a0cb22f","Type":"ContainerStarted","Data":"04eb34f80780cf0341548d1212889380d3f91ccf381f100093151ea1db796e7c"} Aug 04 00:12:41.656617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.656588 2571 generic.go:356] "Generic (PLEG): container finished" podID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerID="ec32be2cca0426b1cfa5080f976133fb90d41938f498273a34c81d8c002e7774" exitCode=0 Aug 04 00:12:41.656730 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.656681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" event={"ID":"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a","Type":"ContainerDied","Data":"ec32be2cca0426b1cfa5080f976133fb90d41938f498273a34c81d8c002e7774"} Aug 04 00:12:41.656800 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.656729 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" event={"ID":"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a","Type":"ContainerStarted","Data":"85b79daf2c1fdb066a9a9ea6ff56f9047c57f8495ec5e2a8ebd5fef9c9868cc1"} Aug 04 00:12:41.658455 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.658429 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerID="fe35c4b8b3e8c8bf39c79e7dd1e691b97486fe533311163e9f9a8bbf41787070" exitCode=0 Aug 04 00:12:41.658565 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.658474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" event={"ID":"6d09f7f5-d7bc-4e17-9738-278eb8564ae1","Type":"ContainerDied","Data":"fe35c4b8b3e8c8bf39c79e7dd1e691b97486fe533311163e9f9a8bbf41787070"} Aug 04 00:12:41.658565 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:41.658510 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" event={"ID":"6d09f7f5-d7bc-4e17-9738-278eb8564ae1","Type":"ContainerStarted","Data":"e534603f954d472305bc5607dc7eee0762af04363b3d0f2769d2f5c5bbc16bbb"} Aug 04 00:12:42.665758 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.665726 2571 generic.go:356] "Generic (PLEG): container finished" podID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerID="b06944a2a4d41ef723a0e883c60524cad6c0647c29a4e4ef8edb60ad872b6585" exitCode=0 Aug 04 00:12:42.666205 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.665806 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" event={"ID":"17988a8e-99f5-4096-be7d-7fee2a0cb22f","Type":"ContainerDied","Data":"b06944a2a4d41ef723a0e883c60524cad6c0647c29a4e4ef8edb60ad872b6585"} Aug 04 00:12:42.667906 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.667880 2571 generic.go:356] "Generic (PLEG): container finished" podID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerID="ac11940f2291ebf728fb784229cd261287c63d43ae7703b986c20524179cfd72" exitCode=0 Aug 04 00:12:42.668014 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.667965 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" event={"ID":"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a","Type":"ContainerDied","Data":"ac11940f2291ebf728fb784229cd261287c63d43ae7703b986c20524179cfd72"} Aug 04 00:12:42.670072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.670050 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" event={"ID":"6d09f7f5-d7bc-4e17-9738-278eb8564ae1","Type":"ContainerStarted","Data":"31dc00d04a9294a27d60c4936bf01060eccdef415c10099b1f9a1cd5bbd4fc30"} Aug 04 00:12:42.671617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.671593 2571 generic.go:356] "Generic (PLEG): container finished" podID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerID="0487076c04b1c5f736a4ca1bad4535c1563c1b3f33b0fa2db5e9a54dd077ef9e" exitCode=0 Aug 04 00:12:42.671718 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:42.671635 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" event={"ID":"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf","Type":"ContainerDied","Data":"0487076c04b1c5f736a4ca1bad4535c1563c1b3f33b0fa2db5e9a54dd077ef9e"} Aug 04 00:12:43.678602 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.678576 2571 generic.go:356] "Generic (PLEG): container finished" podID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerID="5286bd50ecd3309054b4acdfc9ca61b998d55d5a20dd15f221cdddfba0949cd9" exitCode=0 Aug 04 00:12:43.678920 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.678663 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" event={"ID":"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a","Type":"ContainerDied","Data":"5286bd50ecd3309054b4acdfc9ca61b998d55d5a20dd15f221cdddfba0949cd9"} Aug 04 00:12:43.680632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.680610 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerID="31dc00d04a9294a27d60c4936bf01060eccdef415c10099b1f9a1cd5bbd4fc30" exitCode=0 Aug 04 00:12:43.680632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.680628 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerID="752a78a552782c0ae73c5b72fd15e678433f29bb066f29bfd7e1d1f72d5ff271" exitCode=0 Aug 04 00:12:43.680767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.680690 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" event={"ID":"6d09f7f5-d7bc-4e17-9738-278eb8564ae1","Type":"ContainerDied","Data":"31dc00d04a9294a27d60c4936bf01060eccdef415c10099b1f9a1cd5bbd4fc30"} Aug 04 00:12:43.680767 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.680735 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" event={"ID":"6d09f7f5-d7bc-4e17-9738-278eb8564ae1","Type":"ContainerDied","Data":"752a78a552782c0ae73c5b72fd15e678433f29bb066f29bfd7e1d1f72d5ff271"} Aug 04 00:12:43.682305 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.682283 2571 generic.go:356] "Generic (PLEG): container finished" podID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerID="3dc66fffa2ae2b80bf301a907ec42b2562e102aa0f30cfbda1392aa23122f272" exitCode=0 Aug 04 00:12:43.682387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:43.682365 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" event={"ID":"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf","Type":"ContainerDied","Data":"3dc66fffa2ae2b80bf301a907ec42b2562e102aa0f30cfbda1392aa23122f272"} Aug 04 00:12:44.689295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.689259 2571 generic.go:356] "Generic (PLEG): container finished" podID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerID="f37c07dc7f07bde5c84c48ab7cd4b234400ef9b68173fd4dcff311bede0a5656" exitCode=0 Aug 04 00:12:44.691580 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.689341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" event={"ID":"17988a8e-99f5-4096-be7d-7fee2a0cb22f","Type":"ContainerDied","Data":"f37c07dc7f07bde5c84c48ab7cd4b234400ef9b68173fd4dcff311bede0a5656"} Aug 04 00:12:44.691580 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.691500 2571 generic.go:356] "Generic (PLEG): container finished" podID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerID="473d4d07ceb124f256badd0aa3ea5812a918320129501281ab13b76e3a7e2fd2" exitCode=0 Aug 04 00:12:44.692028 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.691579 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" event={"ID":"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf","Type":"ContainerDied","Data":"473d4d07ceb124f256badd0aa3ea5812a918320129501281ab13b76e3a7e2fd2"} Aug 04 00:12:44.832936 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.832907 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:44.851233 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.851210 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:44.922726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.922693 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-util\") pod \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " Aug 04 00:12:44.922899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.922753 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mt45\" (UniqueName: \"kubernetes.io/projected/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-kube-api-access-4mt45\") pod \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " Aug 04 00:12:44.922899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.922780 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jvcf\" (UniqueName: \"kubernetes.io/projected/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-kube-api-access-9jvcf\") pod \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " Aug 04 00:12:44.922899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.922807 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-bundle\") pod \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " Aug 04 00:12:44.923067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.922961 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-util\") pod \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\" (UID: \"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a\") " Aug 04 00:12:44.923067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.923041 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-bundle\") pod \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\" (UID: \"6d09f7f5-d7bc-4e17-9738-278eb8564ae1\") " Aug 04 00:12:44.923592 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.923559 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-bundle" (OuterVolumeSpecName: "bundle") pod "6d09f7f5-d7bc-4e17-9738-278eb8564ae1" (UID: "6d09f7f5-d7bc-4e17-9738-278eb8564ae1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:44.923703 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.923567 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-bundle" (OuterVolumeSpecName: "bundle") pod "ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" (UID: "ae66ba65-c45e-4c4f-a17c-2a09cf21a70a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:44.925388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.925349 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-kube-api-access-4mt45" (OuterVolumeSpecName: "kube-api-access-4mt45") pod "ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" (UID: "ae66ba65-c45e-4c4f-a17c-2a09cf21a70a"). InnerVolumeSpecName "kube-api-access-4mt45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:12:44.925388 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.925381 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-kube-api-access-9jvcf" (OuterVolumeSpecName: "kube-api-access-9jvcf") pod "6d09f7f5-d7bc-4e17-9738-278eb8564ae1" (UID: "6d09f7f5-d7bc-4e17-9738-278eb8564ae1"). InnerVolumeSpecName "kube-api-access-9jvcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:12:44.928795 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.928768 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-util" (OuterVolumeSpecName: "util") pod "6d09f7f5-d7bc-4e17-9738-278eb8564ae1" (UID: "6d09f7f5-d7bc-4e17-9738-278eb8564ae1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:44.928894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:44.928805 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-util" (OuterVolumeSpecName: "util") pod "ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" (UID: "ae66ba65-c45e-4c4f-a17c-2a09cf21a70a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:45.024699 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.024671 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:45.024699 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.024699 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mt45\" (UniqueName: \"kubernetes.io/projected/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-kube-api-access-4mt45\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:45.024886 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.024713 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvcf\" (UniqueName: \"kubernetes.io/projected/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-kube-api-access-9jvcf\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:45.024886 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.024725 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:45.024886 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.024736 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae66ba65-c45e-4c4f-a17c-2a09cf21a70a-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:45.024886 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.024746 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d09f7f5-d7bc-4e17-9738-278eb8564ae1-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:45.700875 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.700836 2571 generic.go:356] "Generic (PLEG): container finished" podID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerID="3c4aaa8ccc4cca0c535be4394ba3e9744c910bf891dc08241e58943a453cf084" exitCode=0 Aug 04 00:12:45.701319 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.700922 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" event={"ID":"17988a8e-99f5-4096-be7d-7fee2a0cb22f","Type":"ContainerDied","Data":"3c4aaa8ccc4cca0c535be4394ba3e9744c910bf891dc08241e58943a453cf084"} Aug 04 00:12:45.702705 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.702683 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" event={"ID":"ae66ba65-c45e-4c4f-a17c-2a09cf21a70a","Type":"ContainerDied","Data":"85b79daf2c1fdb066a9a9ea6ff56f9047c57f8495ec5e2a8ebd5fef9c9868cc1"} Aug 04 00:12:45.702832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.702721 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b79daf2c1fdb066a9a9ea6ff56f9047c57f8495ec5e2a8ebd5fef9c9868cc1" Aug 04 00:12:45.702832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.702694 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053z7j7n" Aug 04 00:12:45.704555 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.704522 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" Aug 04 00:12:45.704664 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.704553 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb868lr" event={"ID":"6d09f7f5-d7bc-4e17-9738-278eb8564ae1","Type":"ContainerDied","Data":"e534603f954d472305bc5607dc7eee0762af04363b3d0f2769d2f5c5bbc16bbb"} Aug 04 00:12:45.704664 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.704605 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e534603f954d472305bc5607dc7eee0762af04363b3d0f2769d2f5c5bbc16bbb" Aug 04 00:12:45.835204 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.835173 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:45.931898 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.931868 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-bundle\") pod \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " Aug 04 00:12:45.932053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.931977 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf4jh\" (UniqueName: \"kubernetes.io/projected/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-kube-api-access-wf4jh\") pod \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " Aug 04 00:12:45.932053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.932003 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-util\") pod \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\" (UID: \"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf\") " Aug 04 00:12:45.932448 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.932414 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-bundle" (OuterVolumeSpecName: "bundle") pod "5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" (UID: "5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:45.934039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.934014 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-kube-api-access-wf4jh" (OuterVolumeSpecName: "kube-api-access-wf4jh") pod "5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" (UID: "5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf"). InnerVolumeSpecName "kube-api-access-wf4jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:12:45.936788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:45.936767 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-util" (OuterVolumeSpecName: "util") pod "5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" (UID: "5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:46.032981 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.032941 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4jh\" (UniqueName: \"kubernetes.io/projected/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-kube-api-access-wf4jh\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:46.032981 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.032983 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:46.033190 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.032994 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:46.711620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.711546 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" Aug 04 00:12:46.711620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.711583 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cw4d6h" event={"ID":"5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf","Type":"ContainerDied","Data":"d71e28b5f9c0ee29f3653ce9e63b603604deabae5f72755b089445694258bf7f"} Aug 04 00:12:46.712062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.711632 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71e28b5f9c0ee29f3653ce9e63b603604deabae5f72755b089445694258bf7f" Aug 04 00:12:46.841908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.841883 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:46.939879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.939846 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-bundle\") pod \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " Aug 04 00:12:46.940036 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.939902 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vqzs\" (UniqueName: \"kubernetes.io/projected/17988a8e-99f5-4096-be7d-7fee2a0cb22f-kube-api-access-7vqzs\") pod \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " Aug 04 00:12:46.940036 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.939959 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-util\") pod \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\" (UID: \"17988a8e-99f5-4096-be7d-7fee2a0cb22f\") " Aug 04 00:12:46.940406 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.940381 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-bundle" (OuterVolumeSpecName: "bundle") pod "17988a8e-99f5-4096-be7d-7fee2a0cb22f" (UID: "17988a8e-99f5-4096-be7d-7fee2a0cb22f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:46.942123 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.942079 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17988a8e-99f5-4096-be7d-7fee2a0cb22f-kube-api-access-7vqzs" (OuterVolumeSpecName: "kube-api-access-7vqzs") pod "17988a8e-99f5-4096-be7d-7fee2a0cb22f" (UID: "17988a8e-99f5-4096-be7d-7fee2a0cb22f"). InnerVolumeSpecName "kube-api-access-7vqzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:12:46.945489 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:46.945462 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-util" (OuterVolumeSpecName: "util") pod "17988a8e-99f5-4096-be7d-7fee2a0cb22f" (UID: "17988a8e-99f5-4096-be7d-7fee2a0cb22f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 00:12:47.041638 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:47.041610 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:47.041638 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:47.041638 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqzs\" (UniqueName: \"kubernetes.io/projected/17988a8e-99f5-4096-be7d-7fee2a0cb22f-kube-api-access-7vqzs\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:47.041817 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:47.041648 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17988a8e-99f5-4096-be7d-7fee2a0cb22f-util\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:47.718714 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:47.718687 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" Aug 04 00:12:47.719075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:47.718727 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0fzxjb" event={"ID":"17988a8e-99f5-4096-be7d-7fee2a0cb22f","Type":"ContainerDied","Data":"04eb34f80780cf0341548d1212889380d3f91ccf381f100093151ea1db796e7c"} Aug 04 00:12:47.719075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:47.718767 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04eb34f80780cf0341548d1212889380d3f91ccf381f100093151ea1db796e7c" Aug 04 00:12:49.642680 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.642638 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57cb95b47-jxk2w" podUID="345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" containerName="console" containerID="cri-o://4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0" gracePeriod=15 Aug 04 00:12:49.895959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.895903 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:12:49.966203 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966176 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-oauth-serving-cert\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966214 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-trusted-ca-bundle\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966251 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-oauth-config\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966266 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49zdh\" (UniqueName: \"kubernetes.io/projected/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-kube-api-access-49zdh\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966291 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-service-ca\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966329 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-config\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966356 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-serving-cert\") pod \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\" (UID: \"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a\") " Aug 04 00:12:49.966781 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966749 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:12:49.966781 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966761 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-service-ca" (OuterVolumeSpecName: "service-ca") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:12:49.966899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966837 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-config" (OuterVolumeSpecName: "console-config") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:12:49.966899 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.966849 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:12:49.968451 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.968424 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-kube-api-access-49zdh" (OuterVolumeSpecName: "kube-api-access-49zdh") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "kube-api-access-49zdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:12:49.968581 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.968561 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:12:49.968634 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:49.968618 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" (UID: "345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:12:50.067272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067234 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-oauth-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.067272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067266 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zdh\" (UniqueName: \"kubernetes.io/projected/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-kube-api-access-49zdh\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.067272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067277 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.067488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067286 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-config\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.067488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067294 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-console-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.067488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067303 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-oauth-serving-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.067488 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.067311 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a-trusted-ca-bundle\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:12:50.733994 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.733904 2571 generic.go:356] "Generic (PLEG): container finished" podID="345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" containerID="4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0" exitCode=0 Aug 04 00:12:50.733994 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.733962 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cb95b47-jxk2w" Aug 04 00:12:50.734463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.733987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb95b47-jxk2w" event={"ID":"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a","Type":"ContainerDied","Data":"4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0"} Aug 04 00:12:50.734463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.734039 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cb95b47-jxk2w" event={"ID":"345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a","Type":"ContainerDied","Data":"c777d93db46300e150b6657be5bd027857fe09cb37273ac8e1e4785c4b05ea95"} Aug 04 00:12:50.734463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.734060 2571 scope.go:117] "RemoveContainer" containerID="4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0" Aug 04 00:12:50.742658 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.742643 2571 scope.go:117] "RemoveContainer" containerID="4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0" Aug 04 00:12:50.742948 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:12:50.742924 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0\": container with ID starting with 4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0 not found: ID does not exist" containerID="4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0" Aug 04 00:12:50.743010 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.742960 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0"} err="failed to get container status \"4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0\": rpc error: code = NotFound desc = could not find container \"4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0\": container with ID starting with 4d1fa7f83d18b085d020990926bbf7257944b1991e93c5cf0c1f7d897de847e0 not found: ID does not exist" Aug 04 00:12:50.759418 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.759393 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57cb95b47-jxk2w"] Aug 04 00:12:50.763532 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:50.763509 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57cb95b47-jxk2w"] Aug 04 00:12:52.378032 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:12:52.377999 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" path="/var/lib/kubelet/pods/345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a/volumes" Aug 04 00:13:59.709599 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709562 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709893 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709905 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709921 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709928 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709934 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709939 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709949 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709954 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709960 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709966 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709971 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709977 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709984 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.709992 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710004 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" containerName="console" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710010 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" containerName="console" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710019 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710026 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710033 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710038 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710044 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710049 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="extract" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710059 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710063 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="pull" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710077 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="util" Aug 04 00:13:59.710074 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710085 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="util" Aug 04 00:13:59.710876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710155 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="345570f1-6d6d-4b0c-b7d7-d8d1a6cea15a" containerName="console" Aug 04 00:13:59.710876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710166 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d09f7f5-d7bc-4e17-9738-278eb8564ae1" containerName="extract" Aug 04 00:13:59.710876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710173 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e5a3023-1391-4b03-9f18-2fa3ad4a8ddf" containerName="extract" Aug 04 00:13:59.710876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710179 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae66ba65-c45e-4c4f-a17c-2a09cf21a70a" containerName="extract" Aug 04 00:13:59.710876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.710187 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="17988a8e-99f5-4096-be7d-7fee2a0cb22f" containerName="extract" Aug 04 00:13:59.714389 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.714369 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:13:59.716860 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.716841 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 00:13:59.716860 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.716856 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 00:13:59.717025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.716844 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rlnxv\"" Aug 04 00:13:59.717425 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.717397 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 00:13:59.720866 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.720841 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:13:59.807751 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.807716 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:13:59.833723 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.833690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8d5\" (UniqueName: \"kubernetes.io/projected/b616d590-0258-4e49-bfca-77ab98d15ea4-kube-api-access-lj8d5\") pod \"limitador-limitador-859c4d96d6-fplk5\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:13:59.833723 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.833728 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b616d590-0258-4e49-bfca-77ab98d15ea4-config-file\") pod \"limitador-limitador-859c4d96d6-fplk5\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:13:59.935048 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.935003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8d5\" (UniqueName: \"kubernetes.io/projected/b616d590-0258-4e49-bfca-77ab98d15ea4-kube-api-access-lj8d5\") pod \"limitador-limitador-859c4d96d6-fplk5\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:13:59.935048 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.935052 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b616d590-0258-4e49-bfca-77ab98d15ea4-config-file\") pod \"limitador-limitador-859c4d96d6-fplk5\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:13:59.935771 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.935752 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b616d590-0258-4e49-bfca-77ab98d15ea4-config-file\") pod \"limitador-limitador-859c4d96d6-fplk5\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:13:59.943011 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:13:59.942975 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8d5\" (UniqueName: \"kubernetes.io/projected/b616d590-0258-4e49-bfca-77ab98d15ea4-kube-api-access-lj8d5\") pod \"limitador-limitador-859c4d96d6-fplk5\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:14:00.025871 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:00.025835 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:14:00.162311 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:00.162283 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:14:00.164603 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:14:00.164572 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb616d590_0258_4e49_bfca_77ab98d15ea4.slice/crio-e87d6177c7d909fe4e9d7a155fcd571fd25c5ad419863a36515d276d1ea6da73 WatchSource:0}: Error finding container e87d6177c7d909fe4e9d7a155fcd571fd25c5ad419863a36515d276d1ea6da73: Status 404 returned error can't find the container with id e87d6177c7d909fe4e9d7a155fcd571fd25c5ad419863a36515d276d1ea6da73 Aug 04 00:14:01.070077 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:01.070040 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" event={"ID":"b616d590-0258-4e49-bfca-77ab98d15ea4","Type":"ContainerStarted","Data":"e87d6177c7d909fe4e9d7a155fcd571fd25c5ad419863a36515d276d1ea6da73"} Aug 04 00:14:11.121364 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:11.121319 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" event={"ID":"b616d590-0258-4e49-bfca-77ab98d15ea4","Type":"ContainerStarted","Data":"e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e"} Aug 04 00:14:11.121764 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:11.121434 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:14:11.137695 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:11.137644 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" podStartSLOduration=1.640571529 podStartE2EDuration="12.137626968s" totalImagesPullingTime="10.497055439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:11.135864332 +0000 UTC m=+689.306897641" watchObservedRunningTime="2026-08-04 00:14:11.137626968 +0000 UTC m=+689.308660274" Aug 04 00:14:14.854664 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:14.854625 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:14:14.855142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:14.854872 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" podUID="b616d590-0258-4e49-bfca-77ab98d15ea4" containerName="limitador" containerID="cri-o://e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e" gracePeriod=30 Aug 04 00:14:15.400179 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.400156 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:14:15.572956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.572917 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b616d590-0258-4e49-bfca-77ab98d15ea4-config-file\") pod \"b616d590-0258-4e49-bfca-77ab98d15ea4\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " Aug 04 00:14:15.573132 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.572989 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj8d5\" (UniqueName: \"kubernetes.io/projected/b616d590-0258-4e49-bfca-77ab98d15ea4-kube-api-access-lj8d5\") pod \"b616d590-0258-4e49-bfca-77ab98d15ea4\" (UID: \"b616d590-0258-4e49-bfca-77ab98d15ea4\") " Aug 04 00:14:15.573394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.573373 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b616d590-0258-4e49-bfca-77ab98d15ea4-config-file" (OuterVolumeSpecName: "config-file") pod "b616d590-0258-4e49-bfca-77ab98d15ea4" (UID: "b616d590-0258-4e49-bfca-77ab98d15ea4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:14:15.575258 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.575238 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616d590-0258-4e49-bfca-77ab98d15ea4-kube-api-access-lj8d5" (OuterVolumeSpecName: "kube-api-access-lj8d5") pod "b616d590-0258-4e49-bfca-77ab98d15ea4" (UID: "b616d590-0258-4e49-bfca-77ab98d15ea4"). InnerVolumeSpecName "kube-api-access-lj8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:14:15.674007 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.673976 2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b616d590-0258-4e49-bfca-77ab98d15ea4-config-file\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:14:15.674007 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:15.674002 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8d5\" (UniqueName: \"kubernetes.io/projected/b616d590-0258-4e49-bfca-77ab98d15ea4-kube-api-access-lj8d5\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:14:16.148027 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.147993 2571 generic.go:356] "Generic (PLEG): container finished" podID="b616d590-0258-4e49-bfca-77ab98d15ea4" containerID="e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e" exitCode=0 Aug 04 00:14:16.148471 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.148036 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" event={"ID":"b616d590-0258-4e49-bfca-77ab98d15ea4","Type":"ContainerDied","Data":"e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e"} Aug 04 00:14:16.148471 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.148039 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" Aug 04 00:14:16.148471 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.148077 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fplk5" event={"ID":"b616d590-0258-4e49-bfca-77ab98d15ea4","Type":"ContainerDied","Data":"e87d6177c7d909fe4e9d7a155fcd571fd25c5ad419863a36515d276d1ea6da73"} Aug 04 00:14:16.148471 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.148129 2571 scope.go:117] "RemoveContainer" containerID="e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e" Aug 04 00:14:16.157621 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.157599 2571 scope.go:117] "RemoveContainer" containerID="e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e" Aug 04 00:14:16.157876 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:14:16.157857 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e\": container with ID starting with e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e not found: ID does not exist" containerID="e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e" Aug 04 00:14:16.157922 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.157884 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e"} err="failed to get container status \"e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e\": rpc error: code = NotFound desc = could not find container \"e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e\": container with ID starting with e2c3e2b2e98e21b27a038f64963b5bc1025a6d6c2c6f08dbc11c79a38b4f7e7e not found: ID does not exist" Aug 04 00:14:16.168824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.168793 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:14:16.170773 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.170756 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fplk5"] Aug 04 00:14:16.382136 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:16.382083 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b616d590-0258-4e49-bfca-77ab98d15ea4" path="/var/lib/kubelet/pods/b616d590-0258-4e49-bfca-77ab98d15ea4/volumes" Aug 04 00:14:21.335370 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.335332 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lffqj"] Aug 04 00:14:21.335868 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.335668 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b616d590-0258-4e49-bfca-77ab98d15ea4" containerName="limitador" Aug 04 00:14:21.335868 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.335680 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b616d590-0258-4e49-bfca-77ab98d15ea4" containerName="limitador" Aug 04 00:14:21.335868 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.335755 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b616d590-0258-4e49-bfca-77ab98d15ea4" containerName="limitador" Aug 04 00:14:21.340231 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.340212 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.342319 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.342296 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 00:14:21.342431 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.342406 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Aug 04 00:14:21.342481 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.342407 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 00:14:21.343198 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.343173 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bmfxk\"" Aug 04 00:14:21.349084 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.349065 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lffqj"] Aug 04 00:14:21.419175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.419136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78548bf1-7c9c-41b2-b775-ae06a31f43a4-data\") pod \"postgres-69fb4594cf-lffqj\" (UID: \"78548bf1-7c9c-41b2-b775-ae06a31f43a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.419175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.419174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57sl\" (UniqueName: \"kubernetes.io/projected/78548bf1-7c9c-41b2-b775-ae06a31f43a4-kube-api-access-p57sl\") pod \"postgres-69fb4594cf-lffqj\" (UID: \"78548bf1-7c9c-41b2-b775-ae06a31f43a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.519927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.519894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78548bf1-7c9c-41b2-b775-ae06a31f43a4-data\") pod \"postgres-69fb4594cf-lffqj\" (UID: \"78548bf1-7c9c-41b2-b775-ae06a31f43a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.519927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.519932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57sl\" (UniqueName: \"kubernetes.io/projected/78548bf1-7c9c-41b2-b775-ae06a31f43a4-kube-api-access-p57sl\") pod \"postgres-69fb4594cf-lffqj\" (UID: \"78548bf1-7c9c-41b2-b775-ae06a31f43a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.520354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.520332 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78548bf1-7c9c-41b2-b775-ae06a31f43a4-data\") pod \"postgres-69fb4594cf-lffqj\" (UID: \"78548bf1-7c9c-41b2-b775-ae06a31f43a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.529282 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.529259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57sl\" (UniqueName: \"kubernetes.io/projected/78548bf1-7c9c-41b2-b775-ae06a31f43a4-kube-api-access-p57sl\") pod \"postgres-69fb4594cf-lffqj\" (UID: \"78548bf1-7c9c-41b2-b775-ae06a31f43a4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.651159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.651047 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:21.780750 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:21.780689 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lffqj"] Aug 04 00:14:21.783011 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:14:21.782976 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78548bf1_7c9c_41b2_b775_ae06a31f43a4.slice/crio-5bdace26f741b53529311c62f3ae0e7ea24d4dbf374c093e2dc073c23aac18ff WatchSource:0}: Error finding container 5bdace26f741b53529311c62f3ae0e7ea24d4dbf374c093e2dc073c23aac18ff: Status 404 returned error can't find the container with id 5bdace26f741b53529311c62f3ae0e7ea24d4dbf374c093e2dc073c23aac18ff Aug 04 00:14:22.179849 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:22.179814 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" event={"ID":"78548bf1-7c9c-41b2-b775-ae06a31f43a4","Type":"ContainerStarted","Data":"5bdace26f741b53529311c62f3ae0e7ea24d4dbf374c093e2dc073c23aac18ff"} Aug 04 00:14:27.208546 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:27.208448 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" event={"ID":"78548bf1-7c9c-41b2-b775-ae06a31f43a4","Type":"ContainerStarted","Data":"ce11372feb22876d9b4f83ebecc97f0bd7669c69649690a197bf4c7335273a4d"} Aug 04 00:14:27.209018 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:27.208579 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:27.224662 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:27.224598 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" podStartSLOduration=1.095772978 podStartE2EDuration="6.224582425s" totalImagesPullingTime="5.128809447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:27.222946224 +0000 UTC m=+705.393979530" watchObservedRunningTime="2026-08-04 00:14:27.224582425 +0000 UTC m=+705.395615732" Aug 04 00:14:33.242297 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:33.242267 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lffqj" Aug 04 00:14:40.263545 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.263458 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-964997bbc-9pk5x"] Aug 04 00:14:40.268380 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.268364 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.272423 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.272389 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-prwj9\"" Aug 04 00:14:40.272897 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.272870 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 00:14:40.273136 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.273090 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 00:14:40.279802 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.279779 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-964997bbc-9pk5x"] Aug 04 00:14:40.378543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.378514 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8jc\" (UniqueName: \"kubernetes.io/projected/92870857-81c1-4e27-b7cc-a3f4bc45729f-kube-api-access-xt8jc\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.378708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.378574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.435411 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.435375 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-f95pn"] Aug 04 00:14:40.437892 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.437874 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.449183 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.449159 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-f95pn"] Aug 04 00:14:40.479198 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.479174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abfb42b0-8875-466a-a549-fd25584a3220-webhook-cert\") pod \"maas-controller-b6cbdb5f9-f95pn\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.479337 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.479236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8jc\" (UniqueName: \"kubernetes.io/projected/92870857-81c1-4e27-b7cc-a3f4bc45729f-kube-api-access-xt8jc\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.479337 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.479277 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cj6k\" (UniqueName: \"kubernetes.io/projected/abfb42b0-8875-466a-a549-fd25584a3220-kube-api-access-9cj6k\") pod \"maas-controller-b6cbdb5f9-f95pn\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.479337 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.479309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.479492 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:14:40.479395 2571 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 00:14:40.479492 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:14:40.479455 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert podName:92870857-81c1-4e27-b7cc-a3f4bc45729f nodeName:}" failed. No retries permitted until 2026-08-04 00:14:40.979438926 +0000 UTC m=+719.150472211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert") pod "maas-controller-964997bbc-9pk5x" (UID: "92870857-81c1-4e27-b7cc-a3f4bc45729f") : secret "maas-controller-webhook-cert" not found Aug 04 00:14:40.490260 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.490239 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8jc\" (UniqueName: \"kubernetes.io/projected/92870857-81c1-4e27-b7cc-a3f4bc45729f-kube-api-access-xt8jc\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.580452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.580411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cj6k\" (UniqueName: \"kubernetes.io/projected/abfb42b0-8875-466a-a549-fd25584a3220-kube-api-access-9cj6k\") pod \"maas-controller-b6cbdb5f9-f95pn\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.580653 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.580569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abfb42b0-8875-466a-a549-fd25584a3220-webhook-cert\") pod \"maas-controller-b6cbdb5f9-f95pn\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.589493 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.589458 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abfb42b0-8875-466a-a549-fd25584a3220-webhook-cert\") pod \"maas-controller-b6cbdb5f9-f95pn\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.589635 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.589469 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cj6k\" (UniqueName: \"kubernetes.io/projected/abfb42b0-8875-466a-a549-fd25584a3220-kube-api-access-9cj6k\") pod \"maas-controller-b6cbdb5f9-f95pn\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.748396 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.748345 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:40.879037 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.879012 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-f95pn"] Aug 04 00:14:40.880853 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:14:40.880825 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfb42b0_8875_466a_a549_fd25584a3220.slice/crio-c44b64b0e52dfd65e26a7532df620dcad56a3de04bb86c6e045650ea787748e0 WatchSource:0}: Error finding container c44b64b0e52dfd65e26a7532df620dcad56a3de04bb86c6e045650ea787748e0: Status 404 returned error can't find the container with id c44b64b0e52dfd65e26a7532df620dcad56a3de04bb86c6e045650ea787748e0 Aug 04 00:14:40.982918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.982885 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:40.985348 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:40.985322 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert\") pod \"maas-controller-964997bbc-9pk5x\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:41.179583 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:41.179497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:41.283534 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:41.283499 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" event={"ID":"abfb42b0-8875-466a-a549-fd25584a3220","Type":"ContainerStarted","Data":"c44b64b0e52dfd65e26a7532df620dcad56a3de04bb86c6e045650ea787748e0"} Aug 04 00:14:41.306918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:41.306894 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-964997bbc-9pk5x"] Aug 04 00:14:41.308625 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:14:41.308603 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92870857_81c1_4e27_b7cc_a3f4bc45729f.slice/crio-e73c5d85ab5261c23ca824a4bb1e2ef31459c3a4de8bf85616273bec3c9afc17 WatchSource:0}: Error finding container e73c5d85ab5261c23ca824a4bb1e2ef31459c3a4de8bf85616273bec3c9afc17: Status 404 returned error can't find the container with id e73c5d85ab5261c23ca824a4bb1e2ef31459c3a4de8bf85616273bec3c9afc17 Aug 04 00:14:42.292040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:42.292000 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-964997bbc-9pk5x" event={"ID":"92870857-81c1-4e27-b7cc-a3f4bc45729f","Type":"ContainerStarted","Data":"e73c5d85ab5261c23ca824a4bb1e2ef31459c3a4de8bf85616273bec3c9afc17"} Aug 04 00:14:43.268414 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:43.268389 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 00:14:44.303632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:44.303592 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-964997bbc-9pk5x" event={"ID":"92870857-81c1-4e27-b7cc-a3f4bc45729f","Type":"ContainerStarted","Data":"fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d"} Aug 04 00:14:44.304123 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:44.303753 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:44.305080 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:44.305054 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" event={"ID":"abfb42b0-8875-466a-a549-fd25584a3220","Type":"ContainerStarted","Data":"5e51d120ee9c3178c1adc47c9dd2afab201e2c73157a1c133ae63ff4edcd654a"} Aug 04 00:14:44.305242 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:44.305180 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:44.334523 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:44.334349 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-964997bbc-9pk5x" podStartSLOduration=2.378233212 podStartE2EDuration="4.334331824s" totalImagesPullingTime="1.956098612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:44.331017456 +0000 UTC m=+722.502050764" watchObservedRunningTime="2026-08-04 00:14:44.334331824 +0000 UTC m=+722.505365132" Aug 04 00:14:44.352961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:44.352910 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" podStartSLOduration=1.962054336 podStartE2EDuration="4.352893374s" totalImagesPullingTime="2.390839038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:14:44.350731221 +0000 UTC m=+722.521764566" watchObservedRunningTime="2026-08-04 00:14:44.352893374 +0000 UTC m=+722.523926682" Aug 04 00:14:55.312727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.312695 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:55.313160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.312847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:14:55.373965 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.373930 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-964997bbc-9pk5x"] Aug 04 00:14:55.374168 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.374124 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-964997bbc-9pk5x" podUID="92870857-81c1-4e27-b7cc-a3f4bc45729f" containerName="manager" containerID="cri-o://fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d" gracePeriod=30 Aug 04 00:14:55.620402 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.620373 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:55.710080 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.710046 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert\") pod \"92870857-81c1-4e27-b7cc-a3f4bc45729f\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " Aug 04 00:14:55.710264 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.710170 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt8jc\" (UniqueName: \"kubernetes.io/projected/92870857-81c1-4e27-b7cc-a3f4bc45729f-kube-api-access-xt8jc\") pod \"92870857-81c1-4e27-b7cc-a3f4bc45729f\" (UID: \"92870857-81c1-4e27-b7cc-a3f4bc45729f\") " Aug 04 00:14:55.712226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.712192 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92870857-81c1-4e27-b7cc-a3f4bc45729f-kube-api-access-xt8jc" (OuterVolumeSpecName: "kube-api-access-xt8jc") pod "92870857-81c1-4e27-b7cc-a3f4bc45729f" (UID: "92870857-81c1-4e27-b7cc-a3f4bc45729f"). InnerVolumeSpecName "kube-api-access-xt8jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:14:55.712338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.712278 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "92870857-81c1-4e27-b7cc-a3f4bc45729f" (UID: "92870857-81c1-4e27-b7cc-a3f4bc45729f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:14:55.811620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.811586 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt8jc\" (UniqueName: \"kubernetes.io/projected/92870857-81c1-4e27-b7cc-a3f4bc45729f-kube-api-access-xt8jc\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:14:55.811620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:55.811615 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92870857-81c1-4e27-b7cc-a3f4bc45729f-webhook-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:14:56.366978 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.366941 2571 generic.go:356] "Generic (PLEG): container finished" podID="92870857-81c1-4e27-b7cc-a3f4bc45729f" containerID="fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d" exitCode=0 Aug 04 00:14:56.367403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.366988 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-964997bbc-9pk5x" Aug 04 00:14:56.367403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.367032 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-964997bbc-9pk5x" event={"ID":"92870857-81c1-4e27-b7cc-a3f4bc45729f","Type":"ContainerDied","Data":"fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d"} Aug 04 00:14:56.367403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.367089 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-964997bbc-9pk5x" event={"ID":"92870857-81c1-4e27-b7cc-a3f4bc45729f","Type":"ContainerDied","Data":"e73c5d85ab5261c23ca824a4bb1e2ef31459c3a4de8bf85616273bec3c9afc17"} Aug 04 00:14:56.367403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.367126 2571 scope.go:117] "RemoveContainer" containerID="fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d" Aug 04 00:14:56.376651 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.376428 2571 scope.go:117] "RemoveContainer" containerID="fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d" Aug 04 00:14:56.376727 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:14:56.376665 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d\": container with ID starting with fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d not found: ID does not exist" containerID="fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d" Aug 04 00:14:56.376727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.376689 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d"} err="failed to get container status \"fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d\": rpc error: code = NotFound desc = could not find container \"fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d\": container with ID starting with fd76a120c7affd3ad3137b614cbda5ec0988cf999156c525abb149ea2953845d not found: ID does not exist" Aug 04 00:14:56.391041 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.391015 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-964997bbc-9pk5x"] Aug 04 00:14:56.396381 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:56.396361 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-964997bbc-9pk5x"] Aug 04 00:14:58.376442 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:14:58.376407 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92870857-81c1-4e27-b7cc-a3f4bc45729f" path="/var/lib/kubelet/pods/92870857-81c1-4e27-b7cc-a3f4bc45729f/volumes" Aug 04 00:15:16.793531 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.793497 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67c8968587-lpv6f"] Aug 04 00:15:16.794018 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.793894 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92870857-81c1-4e27-b7cc-a3f4bc45729f" containerName="manager" Aug 04 00:15:16.794018 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.793909 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="92870857-81c1-4e27-b7cc-a3f4bc45729f" containerName="manager" Aug 04 00:15:16.794018 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.793977 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="92870857-81c1-4e27-b7cc-a3f4bc45729f" containerName="manager" Aug 04 00:15:16.801015 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.800993 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.803512 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.803483 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pvrls\"" Aug 04 00:15:16.803865 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.803666 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 00:15:16.806243 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.806222 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c8968587-lpv6f"] Aug 04 00:15:16.891477 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.891441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hz6m\" (UniqueName: \"kubernetes.io/projected/9b0d62de-09ea-471d-8c61-b0b75b1afc87-kube-api-access-2hz6m\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.891643 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.891493 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0d62de-09ea-471d-8c61-b0b75b1afc87-maas-api-tls\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.891643 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.891546 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d62de-09ea-471d-8c61-b0b75b1afc87-openshift-service-ca\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.992002 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.991966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hz6m\" (UniqueName: \"kubernetes.io/projected/9b0d62de-09ea-471d-8c61-b0b75b1afc87-kube-api-access-2hz6m\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.992226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.992024 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0d62de-09ea-471d-8c61-b0b75b1afc87-maas-api-tls\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.992226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.992057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d62de-09ea-471d-8c61-b0b75b1afc87-openshift-service-ca\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:16.992360 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:15:16.992223 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Aug 04 00:15:16.992360 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:15:16.992332 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0d62de-09ea-471d-8c61-b0b75b1afc87-maas-api-tls podName:9b0d62de-09ea-471d-8c61-b0b75b1afc87 nodeName:}" failed. No retries permitted until 2026-08-04 00:15:17.492308738 +0000 UTC m=+755.663342023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b0d62de-09ea-471d-8c61-b0b75b1afc87-maas-api-tls") pod "maas-api-67c8968587-lpv6f" (UID: "9b0d62de-09ea-471d-8c61-b0b75b1afc87") : secret "maas-api-serving-cert" not found Aug 04 00:15:16.992716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:16.992695 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0d62de-09ea-471d-8c61-b0b75b1afc87-openshift-service-ca\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:17.000082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.000056 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hz6m\" (UniqueName: \"kubernetes.io/projected/9b0d62de-09ea-471d-8c61-b0b75b1afc87-kube-api-access-2hz6m\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:17.287351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.287320 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl"] Aug 04 00:15:17.292212 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.292197 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.294529 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.294506 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gn6px\"" Aug 04 00:15:17.294648 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.294538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzmm\" (UniqueName: \"kubernetes.io/projected/6d43038d-b43d-48ec-920e-7df1ddc144d6-kube-api-access-kjzmm\") pod \"payload-pre-processing-687dbb6cf6-njlwl\" (UID: \"6d43038d-b43d-48ec-920e-7df1ddc144d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.294648 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.294590 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43038d-b43d-48ec-920e-7df1ddc144d6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-njlwl\" (UID: \"6d43038d-b43d-48ec-920e-7df1ddc144d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.294648 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.294549 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 00:15:17.298591 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.298565 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl"] Aug 04 00:15:17.395514 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.395480 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzmm\" (UniqueName: \"kubernetes.io/projected/6d43038d-b43d-48ec-920e-7df1ddc144d6-kube-api-access-kjzmm\") pod \"payload-pre-processing-687dbb6cf6-njlwl\" (UID: \"6d43038d-b43d-48ec-920e-7df1ddc144d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.395727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.395530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43038d-b43d-48ec-920e-7df1ddc144d6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-njlwl\" (UID: \"6d43038d-b43d-48ec-920e-7df1ddc144d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.396214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.396193 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43038d-b43d-48ec-920e-7df1ddc144d6-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-njlwl\" (UID: \"6d43038d-b43d-48ec-920e-7df1ddc144d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.403749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.403718 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzmm\" (UniqueName: \"kubernetes.io/projected/6d43038d-b43d-48ec-920e-7df1ddc144d6-kube-api-access-kjzmm\") pod \"payload-pre-processing-687dbb6cf6-njlwl\" (UID: \"6d43038d-b43d-48ec-920e-7df1ddc144d6\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.496630 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.496594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0d62de-09ea-471d-8c61-b0b75b1afc87-maas-api-tls\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:17.499267 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.499234 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0d62de-09ea-471d-8c61-b0b75b1afc87-maas-api-tls\") pod \"maas-api-67c8968587-lpv6f\" (UID: \"9b0d62de-09ea-471d-8c61-b0b75b1afc87\") " pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:17.604667 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.604576 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:17.713090 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.713053 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:17.737695 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.737666 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl"] Aug 04 00:15:17.739088 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:15:17.739063 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d43038d_b43d_48ec_920e_7df1ddc144d6.slice/crio-17a6704e10f5366787cceaa1b4d51a4e3bfb65436928c0f8a6d42e5b7e81eeec WatchSource:0}: Error finding container 17a6704e10f5366787cceaa1b4d51a4e3bfb65436928c0f8a6d42e5b7e81eeec: Status 404 returned error can't find the container with id 17a6704e10f5366787cceaa1b4d51a4e3bfb65436928c0f8a6d42e5b7e81eeec Aug 04 00:15:17.741069 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.741048 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:15:17.854352 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:17.854325 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c8968587-lpv6f"] Aug 04 00:15:17.856719 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:15:17.856659 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0d62de_09ea_471d_8c61_b0b75b1afc87.slice/crio-76c32ff7a596207055e9568644a4e60e0da798fc39d4c7c8a33b136baeeb6462 WatchSource:0}: Error finding container 76c32ff7a596207055e9568644a4e60e0da798fc39d4c7c8a33b136baeeb6462: Status 404 returned error can't find the container with id 76c32ff7a596207055e9568644a4e60e0da798fc39d4c7c8a33b136baeeb6462 Aug 04 00:15:18.482279 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:18.482240 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" event={"ID":"6d43038d-b43d-48ec-920e-7df1ddc144d6","Type":"ContainerStarted","Data":"17a6704e10f5366787cceaa1b4d51a4e3bfb65436928c0f8a6d42e5b7e81eeec"} Aug 04 00:15:18.483443 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:18.483416 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" event={"ID":"9b0d62de-09ea-471d-8c61-b0b75b1afc87","Type":"ContainerStarted","Data":"76c32ff7a596207055e9568644a4e60e0da798fc39d4c7c8a33b136baeeb6462"} Aug 04 00:15:20.495654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:20.495615 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" event={"ID":"9b0d62de-09ea-471d-8c61-b0b75b1afc87","Type":"ContainerStarted","Data":"b6d30287e5191f7d9cb32af1fc490850f4dfd827a822fea03aedcfd6020da39b"} Aug 04 00:15:20.496122 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:20.495771 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:20.497308 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:20.497286 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" event={"ID":"6d43038d-b43d-48ec-920e-7df1ddc144d6","Type":"ContainerStarted","Data":"8609c4cb123570db7f7d3b8b0e5cb5f860179338d6ed7a9c5d1c48c178421b58"} Aug 04 00:15:20.497415 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:20.497348 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:20.512859 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:20.512809 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" podStartSLOduration=2.5273311 podStartE2EDuration="4.512788431s" totalImagesPullingTime="1.985457331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:20.510191047 +0000 UTC m=+758.681224354" watchObservedRunningTime="2026-08-04 00:15:20.512788431 +0000 UTC m=+758.683821738" Aug 04 00:15:20.525192 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:20.525150 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" podStartSLOduration=1.414516648 podStartE2EDuration="3.525136013s" totalImagesPullingTime="2.110619365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:20.524058555 +0000 UTC m=+758.695091873" watchObservedRunningTime="2026-08-04 00:15:20.525136013 +0000 UTC m=+758.696169319" Aug 04 00:15:26.508756 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:26.508728 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67c8968587-lpv6f" Aug 04 00:15:31.504944 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:31.504912 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-njlwl" Aug 04 00:15:32.839723 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.839685 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4"] Aug 04 00:15:32.843696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.843673 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:32.846395 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.846371 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 04 00:15:32.846519 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.846376 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 00:15:32.846978 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.846963 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k7knc\"" Aug 04 00:15:32.847069 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.846993 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 00:15:32.852987 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.852963 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4"] Aug 04 00:15:32.927820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.927787 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7s7\" (UniqueName: \"kubernetes.io/projected/fb1c8d19-9577-44dc-8f21-b23640984439-kube-api-access-vj7s7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:32.927986 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.927838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:32.927986 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.927859 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:32.927986 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.927942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:32.927986 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.927978 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c8d19-9577-44dc-8f21-b23640984439-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:32.928182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:32.928019 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029134 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029331 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c8d19-9577-44dc-8f21-b23640984439-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029331 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029331 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7s7\" (UniqueName: \"kubernetes.io/projected/fb1c8d19-9577-44dc-8f21-b23640984439-kube-api-access-vj7s7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029331 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029331 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029327 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029636 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029606 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029771 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.029820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.029784 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.031546 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.031524 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb1c8d19-9577-44dc-8f21-b23640984439-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.031904 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.031887 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1c8d19-9577-44dc-8f21-b23640984439-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.037373 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.037341 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7s7\" (UniqueName: \"kubernetes.io/projected/fb1c8d19-9577-44dc-8f21-b23640984439-kube-api-access-vj7s7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4\" (UID: \"fb1c8d19-9577-44dc-8f21-b23640984439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.155268 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.155179 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:33.291017 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.290990 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4"] Aug 04 00:15:33.292818 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:15:33.292783 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1c8d19_9577_44dc_8f21_b23640984439.slice/crio-d19720afe2c108be1e744266950322afbc325d740307b7aa37baed915bda3ba2 WatchSource:0}: Error finding container d19720afe2c108be1e744266950322afbc325d740307b7aa37baed915bda3ba2: Status 404 returned error can't find the container with id d19720afe2c108be1e744266950322afbc325d740307b7aa37baed915bda3ba2 Aug 04 00:15:33.567197 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:33.567149 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" event={"ID":"fb1c8d19-9577-44dc-8f21-b23640984439","Type":"ContainerStarted","Data":"d19720afe2c108be1e744266950322afbc325d740307b7aa37baed915bda3ba2"} Aug 04 00:15:34.573920 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:34.573884 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" event={"ID":"fb1c8d19-9577-44dc-8f21-b23640984439","Type":"ContainerStarted","Data":"516f88333df1454e1cacd5517ba536d27f7ad1e51f280786b4ccceaedbc64b7b"} Aug 04 00:15:34.591199 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:34.591128 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" podStartSLOduration=1.470246706 podStartE2EDuration="2.591086892s" totalImagesPullingTime="1.120840186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:34.590259834 +0000 UTC m=+772.761293142" watchObservedRunningTime="2026-08-04 00:15:34.591086892 +0000 UTC m=+772.762120199" Aug 04 00:15:43.155450 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:43.155400 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:43.155931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:43.155477 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:43.167656 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:43.167628 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:43.630441 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:43.630409 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4" Aug 04 00:15:58.921974 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:58.921940 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr"] Aug 04 00:15:58.928010 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:58.927987 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:58.930208 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:58.930182 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 00:15:58.936816 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:58.936789 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr"] Aug 04 00:15:59.070143 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.070092 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48bc7\" (UniqueName: \"kubernetes.io/projected/50b80ba0-4de2-42d8-b1c6-02153a91be21-kube-api-access-48bc7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.070329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.070157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.070329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.070218 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.070329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.070265 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.070329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.070286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b80ba0-4de2-42d8-b1c6-02153a91be21-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.070329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.070326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.170983 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.170935 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171206 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171206 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b80ba0-4de2-42d8-b1c6-02153a91be21-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171206 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171206 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc7\" (UniqueName: \"kubernetes.io/projected/50b80ba0-4de2-42d8-b1c6-02153a91be21-kube-api-access-48bc7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171206 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171200 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171498 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171404 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171498 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171435 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.171594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.171502 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.173504 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.173448 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50b80ba0-4de2-42d8-b1c6-02153a91be21-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.173661 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.173644 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b80ba0-4de2-42d8-b1c6-02153a91be21-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.180500 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.180474 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bc7\" (UniqueName: \"kubernetes.io/projected/50b80ba0-4de2-42d8-b1c6-02153a91be21-kube-api-access-48bc7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr\" (UID: \"50b80ba0-4de2-42d8-b1c6-02153a91be21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.240551 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.240520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:15:59.375230 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.375207 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr"] Aug 04 00:15:59.377126 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:15:59.377064 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b80ba0_4de2_42d8_b1c6_02153a91be21.slice/crio-f84fb6f1e1e35b0974a3540896af551f6faa7c41d692fe9caaf47f77951d7a2e WatchSource:0}: Error finding container f84fb6f1e1e35b0974a3540896af551f6faa7c41d692fe9caaf47f77951d7a2e: Status 404 returned error can't find the container with id f84fb6f1e1e35b0974a3540896af551f6faa7c41d692fe9caaf47f77951d7a2e Aug 04 00:15:59.708038 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.707998 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" event={"ID":"50b80ba0-4de2-42d8-b1c6-02153a91be21","Type":"ContainerStarted","Data":"9343f0d31e391f33ae6983206d79e7385cc842b3dd234ccd15e1366d7f03e0c6"} Aug 04 00:15:59.708221 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:15:59.708056 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" event={"ID":"50b80ba0-4de2-42d8-b1c6-02153a91be21","Type":"ContainerStarted","Data":"f84fb6f1e1e35b0974a3540896af551f6faa7c41d692fe9caaf47f77951d7a2e"} Aug 04 00:16:09.241025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:16:09.240927 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:16:09.241025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:16:09.241014 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:16:09.253867 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:16:09.253840 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:16:09.274584 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:16:09.274531 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" podStartSLOduration=11.08905439 podStartE2EDuration="11.274512261s" totalImagesPullingTime="185.457871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:15:59.724817162 +0000 UTC m=+797.895850494" watchObservedRunningTime="2026-08-04 00:16:09.274512261 +0000 UTC m=+807.445545570" Aug 04 00:16:09.775407 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:16:09.775370 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr" Aug 04 00:18:52.346610 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.346574 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-f95pn"] Aug 04 00:18:52.347065 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.346773 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" podUID="abfb42b0-8875-466a-a549-fd25584a3220" containerName="manager" containerID="cri-o://5e51d120ee9c3178c1adc47c9dd2afab201e2c73157a1c133ae63ff4edcd654a" gracePeriod=30 Aug 04 00:18:52.595828 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.595800 2571 generic.go:356] "Generic (PLEG): container finished" podID="abfb42b0-8875-466a-a549-fd25584a3220" containerID="5e51d120ee9c3178c1adc47c9dd2afab201e2c73157a1c133ae63ff4edcd654a" exitCode=0 Aug 04 00:18:52.595944 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.595859 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" event={"ID":"abfb42b0-8875-466a-a549-fd25584a3220","Type":"ContainerDied","Data":"5e51d120ee9c3178c1adc47c9dd2afab201e2c73157a1c133ae63ff4edcd654a"} Aug 04 00:18:52.595944 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.595897 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" event={"ID":"abfb42b0-8875-466a-a549-fd25584a3220","Type":"ContainerDied","Data":"c44b64b0e52dfd65e26a7532df620dcad56a3de04bb86c6e045650ea787748e0"} Aug 04 00:18:52.595944 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.595913 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44b64b0e52dfd65e26a7532df620dcad56a3de04bb86c6e045650ea787748e0" Aug 04 00:18:52.597876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.597836 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:18:52.657374 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.657339 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cj6k\" (UniqueName: \"kubernetes.io/projected/abfb42b0-8875-466a-a549-fd25584a3220-kube-api-access-9cj6k\") pod \"abfb42b0-8875-466a-a549-fd25584a3220\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " Aug 04 00:18:52.657541 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.657407 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abfb42b0-8875-466a-a549-fd25584a3220-webhook-cert\") pod \"abfb42b0-8875-466a-a549-fd25584a3220\" (UID: \"abfb42b0-8875-466a-a549-fd25584a3220\") " Aug 04 00:18:52.659446 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.659412 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfb42b0-8875-466a-a549-fd25584a3220-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "abfb42b0-8875-466a-a549-fd25584a3220" (UID: "abfb42b0-8875-466a-a549-fd25584a3220"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:18:52.659582 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.659544 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfb42b0-8875-466a-a549-fd25584a3220-kube-api-access-9cj6k" (OuterVolumeSpecName: "kube-api-access-9cj6k") pod "abfb42b0-8875-466a-a549-fd25584a3220" (UID: "abfb42b0-8875-466a-a549-fd25584a3220"). InnerVolumeSpecName "kube-api-access-9cj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:18:52.758214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.758184 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cj6k\" (UniqueName: \"kubernetes.io/projected/abfb42b0-8875-466a-a549-fd25584a3220-kube-api-access-9cj6k\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:18:52.758214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:52.758207 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abfb42b0-8875-466a-a549-fd25584a3220-webhook-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:18:53.601117 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:53.601079 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-f95pn" Aug 04 00:18:53.622273 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:53.622239 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-f95pn"] Aug 04 00:18:53.625173 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:53.625147 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-f95pn"] Aug 04 00:18:54.382354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:18:54.382306 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfb42b0-8875-466a-a549-fd25584a3220" path="/var/lib/kubelet/pods/abfb42b0-8875-466a-a549-fd25584a3220/volumes" Aug 04 00:19:54.472981 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.472947 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-6hsr5"] Aug 04 00:19:54.473507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.473301 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfb42b0-8875-466a-a549-fd25584a3220" containerName="manager" Aug 04 00:19:54.473507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.473312 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfb42b0-8875-466a-a549-fd25584a3220" containerName="manager" Aug 04 00:19:54.473507 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.473378 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfb42b0-8875-466a-a549-fd25584a3220" containerName="manager" Aug 04 00:19:54.476480 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.476461 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.479643 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.479619 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 00:19:54.479643 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.479635 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-prwj9\"" Aug 04 00:19:54.479824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.479623 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 00:19:54.487713 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.487693 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-6hsr5"] Aug 04 00:19:54.586622 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.586590 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgw6\" (UniqueName: \"kubernetes.io/projected/026b0aee-508d-4c04-a29c-c103aec7335e-kube-api-access-7xgw6\") pod \"maas-controller-b6cbdb5f9-6hsr5\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.586799 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.586644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026b0aee-508d-4c04-a29c-c103aec7335e-webhook-cert\") pod \"maas-controller-b6cbdb5f9-6hsr5\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.687578 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.687551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026b0aee-508d-4c04-a29c-c103aec7335e-webhook-cert\") pod \"maas-controller-b6cbdb5f9-6hsr5\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.687723 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.687642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgw6\" (UniqueName: \"kubernetes.io/projected/026b0aee-508d-4c04-a29c-c103aec7335e-kube-api-access-7xgw6\") pod \"maas-controller-b6cbdb5f9-6hsr5\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.690071 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.690051 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026b0aee-508d-4c04-a29c-c103aec7335e-webhook-cert\") pod \"maas-controller-b6cbdb5f9-6hsr5\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.695611 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.695581 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgw6\" (UniqueName: \"kubernetes.io/projected/026b0aee-508d-4c04-a29c-c103aec7335e-kube-api-access-7xgw6\") pod \"maas-controller-b6cbdb5f9-6hsr5\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.786738 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.786706 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:54.919777 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:54.919754 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-6hsr5"] Aug 04 00:19:54.921973 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:19:54.921942 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026b0aee_508d_4c04_a29c_c103aec7335e.slice/crio-2226d3a30eed896bcaa1164b5b735ff18954caf06fa9ba2667abcb37852737c4 WatchSource:0}: Error finding container 2226d3a30eed896bcaa1164b5b735ff18954caf06fa9ba2667abcb37852737c4: Status 404 returned error can't find the container with id 2226d3a30eed896bcaa1164b5b735ff18954caf06fa9ba2667abcb37852737c4 Aug 04 00:19:55.921888 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:55.921846 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" event={"ID":"026b0aee-508d-4c04-a29c-c103aec7335e","Type":"ContainerStarted","Data":"ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf"} Aug 04 00:19:55.922354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:55.921899 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" event={"ID":"026b0aee-508d-4c04-a29c-c103aec7335e","Type":"ContainerStarted","Data":"2226d3a30eed896bcaa1164b5b735ff18954caf06fa9ba2667abcb37852737c4"} Aug 04 00:19:55.922354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:55.921958 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:19:55.939585 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:19:55.939532 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" podStartSLOduration=1.50715541 podStartE2EDuration="1.939517537s" totalImagesPullingTime="432.362127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:19:55.938309487 +0000 UTC m=+1034.109342795" watchObservedRunningTime="2026-08-04 00:19:55.939517537 +0000 UTC m=+1034.110550843" Aug 04 00:20:06.929702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:06.929672 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:20:11.596578 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.596541 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-6hsr5"] Aug 04 00:20:11.596959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.596744 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" podUID="026b0aee-508d-4c04-a29c-c103aec7335e" containerName="manager" containerID="cri-o://ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf" gracePeriod=30 Aug 04 00:20:11.838414 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.838385 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:20:11.940197 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.940126 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026b0aee-508d-4c04-a29c-c103aec7335e-webhook-cert\") pod \"026b0aee-508d-4c04-a29c-c103aec7335e\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " Aug 04 00:20:11.940197 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.940164 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xgw6\" (UniqueName: \"kubernetes.io/projected/026b0aee-508d-4c04-a29c-c103aec7335e-kube-api-access-7xgw6\") pod \"026b0aee-508d-4c04-a29c-c103aec7335e\" (UID: \"026b0aee-508d-4c04-a29c-c103aec7335e\") " Aug 04 00:20:11.942133 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.942084 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026b0aee-508d-4c04-a29c-c103aec7335e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "026b0aee-508d-4c04-a29c-c103aec7335e" (UID: "026b0aee-508d-4c04-a29c-c103aec7335e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:20:11.942258 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:11.942214 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026b0aee-508d-4c04-a29c-c103aec7335e-kube-api-access-7xgw6" (OuterVolumeSpecName: "kube-api-access-7xgw6") pod "026b0aee-508d-4c04-a29c-c103aec7335e" (UID: "026b0aee-508d-4c04-a29c-c103aec7335e"). InnerVolumeSpecName "kube-api-access-7xgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:20:12.009820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.009789 2571 generic.go:356] "Generic (PLEG): container finished" podID="026b0aee-508d-4c04-a29c-c103aec7335e" containerID="ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf" exitCode=0 Aug 04 00:20:12.009989 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.009830 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" Aug 04 00:20:12.009989 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.009877 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" event={"ID":"026b0aee-508d-4c04-a29c-c103aec7335e","Type":"ContainerDied","Data":"ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf"} Aug 04 00:20:12.009989 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.009931 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-6hsr5" event={"ID":"026b0aee-508d-4c04-a29c-c103aec7335e","Type":"ContainerDied","Data":"2226d3a30eed896bcaa1164b5b735ff18954caf06fa9ba2667abcb37852737c4"} Aug 04 00:20:12.009989 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.009960 2571 scope.go:117] "RemoveContainer" containerID="ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf" Aug 04 00:20:12.021075 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.021050 2571 scope.go:117] "RemoveContainer" containerID="ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf" Aug 04 00:20:12.021580 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:20:12.021550 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf\": container with ID starting with ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf not found: ID does not exist" containerID="ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf" Aug 04 00:20:12.021675 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.021594 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf"} err="failed to get container status \"ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf\": rpc error: code = NotFound desc = could not find container \"ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf\": container with ID starting with ea33b26a28e249fc963f3df437b991f51d0a6d5528dba9138ee3304b3ac5d7cf not found: ID does not exist" Aug 04 00:20:12.033956 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.033932 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-6hsr5"] Aug 04 00:20:12.036775 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.036755 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-6hsr5"] Aug 04 00:20:12.040860 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.040843 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026b0aee-508d-4c04-a29c-c103aec7335e-webhook-cert\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:20:12.040924 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.040863 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xgw6\" (UniqueName: \"kubernetes.io/projected/026b0aee-508d-4c04-a29c-c103aec7335e-kube-api-access-7xgw6\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:20:12.377716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.377685 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="026b0aee-508d-4c04-a29c-c103aec7335e" path="/var/lib/kubelet/pods/026b0aee-508d-4c04-a29c-c103aec7335e/volumes" Aug 04 00:20:12.921306 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.921260 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-kgpcj"] Aug 04 00:20:12.921710 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.921694 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026b0aee-508d-4c04-a29c-c103aec7335e" containerName="manager" Aug 04 00:20:12.921755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.921712 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="026b0aee-508d-4c04-a29c-c103aec7335e" containerName="manager" Aug 04 00:20:12.921797 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.921782 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="026b0aee-508d-4c04-a29c-c103aec7335e" containerName="manager" Aug 04 00:20:12.926247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.926229 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:12.928566 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.928538 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 00:20:12.928687 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.928590 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-prwj9\"" Aug 04 00:20:12.928880 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.928866 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 00:20:12.933037 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:12.933016 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-kgpcj"] Aug 04 00:20:13.049281 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.049248 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6805a8f3-adea-4c35-8ef8-c41d0e80b1e5-webhook-cert\") pod \"maas-controller-b6cbdb5f9-kgpcj\" (UID: \"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5\") " pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.049478 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.049302 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb9v9\" (UniqueName: \"kubernetes.io/projected/6805a8f3-adea-4c35-8ef8-c41d0e80b1e5-kube-api-access-lb9v9\") pod \"maas-controller-b6cbdb5f9-kgpcj\" (UID: \"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5\") " pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.149712 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.149676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6805a8f3-adea-4c35-8ef8-c41d0e80b1e5-webhook-cert\") pod \"maas-controller-b6cbdb5f9-kgpcj\" (UID: \"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5\") " pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.149890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.149752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb9v9\" (UniqueName: \"kubernetes.io/projected/6805a8f3-adea-4c35-8ef8-c41d0e80b1e5-kube-api-access-lb9v9\") pod \"maas-controller-b6cbdb5f9-kgpcj\" (UID: \"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5\") " pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.152089 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.152062 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6805a8f3-adea-4c35-8ef8-c41d0e80b1e5-webhook-cert\") pod \"maas-controller-b6cbdb5f9-kgpcj\" (UID: \"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5\") " pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.158049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.158024 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb9v9\" (UniqueName: \"kubernetes.io/projected/6805a8f3-adea-4c35-8ef8-c41d0e80b1e5-kube-api-access-lb9v9\") pod \"maas-controller-b6cbdb5f9-kgpcj\" (UID: \"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5\") " pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.238391 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.238304 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:13.366614 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:13.366580 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b6cbdb5f9-kgpcj"] Aug 04 00:20:13.367908 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:20:13.367881 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6805a8f3_adea_4c35_8ef8_c41d0e80b1e5.slice/crio-bf9c354ca00f434d31867f1b3fb4d9f62e7d5454ffe81565fd6884f0ee664f74 WatchSource:0}: Error finding container bf9c354ca00f434d31867f1b3fb4d9f62e7d5454ffe81565fd6884f0ee664f74: Status 404 returned error can't find the container with id bf9c354ca00f434d31867f1b3fb4d9f62e7d5454ffe81565fd6884f0ee664f74 Aug 04 00:20:14.022698 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:14.022660 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" event={"ID":"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5","Type":"ContainerStarted","Data":"1d31076d9ef568c9c0908936602a105390d44bc980930a6d5fd9e7f3fcbcaf80"} Aug 04 00:20:14.023264 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:14.022745 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" event={"ID":"6805a8f3-adea-4c35-8ef8-c41d0e80b1e5","Type":"ContainerStarted","Data":"bf9c354ca00f434d31867f1b3fb4d9f62e7d5454ffe81565fd6884f0ee664f74"} Aug 04 00:20:14.023264 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:14.022901 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:20:14.041250 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:14.041201 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" podStartSLOduration=1.6930411840000001 podStartE2EDuration="2.041187111s" totalImagesPullingTime="348.145927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:20:14.039838044 +0000 UTC m=+1052.210871349" watchObservedRunningTime="2026-08-04 00:20:14.041187111 +0000 UTC m=+1052.212220418" Aug 04 00:20:25.030515 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:20:25.030482 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b6cbdb5f9-kgpcj" Aug 04 00:21:43.294763 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:21:43.294732 2571 scope.go:117] "RemoveContainer" containerID="5e51d120ee9c3178c1adc47c9dd2afab201e2c73157a1c133ae63ff4edcd654a" Aug 04 00:35:21.887639 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.887606 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v"] Aug 04 00:35:21.891160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.891142 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:21.894250 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.894227 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-418d3ab6-serving-cert\"" Aug 04 00:35:21.902227 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.902198 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v"] Aug 04 00:35:21.917217 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.917192 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258ed20e-98b8-4e2c-b6fe-549517813f79-maas-api-tls\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:21.917323 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.917225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdd9x\" (UniqueName: \"kubernetes.io/projected/258ed20e-98b8-4e2c-b6fe-549517813f79-kube-api-access-sdd9x\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:21.917323 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:21.917294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258ed20e-98b8-4e2c-b6fe-549517813f79-openshift-service-ca\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.017818 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.017777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258ed20e-98b8-4e2c-b6fe-549517813f79-maas-api-tls\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.018012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.017825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdd9x\" (UniqueName: \"kubernetes.io/projected/258ed20e-98b8-4e2c-b6fe-549517813f79-kube-api-access-sdd9x\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.018012 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.017920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258ed20e-98b8-4e2c-b6fe-549517813f79-openshift-service-ca\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.018648 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.018628 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258ed20e-98b8-4e2c-b6fe-549517813f79-openshift-service-ca\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.020665 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.020639 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258ed20e-98b8-4e2c-b6fe-549517813f79-maas-api-tls\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.026533 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.026509 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdd9x\" (UniqueName: \"kubernetes.io/projected/258ed20e-98b8-4e2c-b6fe-549517813f79-kube-api-access-sdd9x\") pod \"maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.204806 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.204722 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:22.345848 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.345816 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v"] Aug 04 00:35:22.347906 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:35:22.347867 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258ed20e_98b8_4e2c_b6fe_549517813f79.slice/crio-60cca6f18731639c45cfb43ee72689a34961fa333924422dbc6d19407bcac7bd WatchSource:0}: Error finding container 60cca6f18731639c45cfb43ee72689a34961fa333924422dbc6d19407bcac7bd: Status 404 returned error can't find the container with id 60cca6f18731639c45cfb43ee72689a34961fa333924422dbc6d19407bcac7bd Aug 04 00:35:22.349164 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.349147 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:35:22.546493 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.546464 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh"] Aug 04 00:35:22.551202 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.551179 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.553454 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.553429 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-418d3ab6\"" Aug 04 00:35:22.553661 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.553637 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-418d3ab6-dockercfg-xhxm4\"" Aug 04 00:35:22.560142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.560120 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh"] Aug 04 00:35:22.623550 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.623515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-kube-api-access-wdbph\") pod \"payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.623721 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.623563 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.681045 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.681012 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" event={"ID":"258ed20e-98b8-4e2c-b6fe-549517813f79","Type":"ContainerStarted","Data":"60cca6f18731639c45cfb43ee72689a34961fa333924422dbc6d19407bcac7bd"} Aug 04 00:35:22.724441 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.724415 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-kube-api-access-wdbph\") pod \"payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.724562 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.724460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.725145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.725129 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.732394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.732378 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-kube-api-access-wdbph\") pod \"payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.864162 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.864071 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:22.996659 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:22.996618 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh"] Aug 04 00:35:23.000132 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:35:23.000093 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08de0f2d_5d41_4ed7_a4e3_2413b16271a5.slice/crio-e1406d2a61a7eaec64c94a4351f8fff2046d99f62c7abaddc92cff846dc8b30d WatchSource:0}: Error finding container e1406d2a61a7eaec64c94a4351f8fff2046d99f62c7abaddc92cff846dc8b30d: Status 404 returned error can't find the container with id e1406d2a61a7eaec64c94a4351f8fff2046d99f62c7abaddc92cff846dc8b30d Aug 04 00:35:23.689168 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:23.689129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" event={"ID":"08de0f2d-5d41-4ed7-a4e3-2413b16271a5","Type":"ContainerStarted","Data":"2063df50a69c39b6c7b47acdab9a4cb3c75572876d1c2e14220ee7dc1282d40c"} Aug 04 00:35:23.689382 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:23.689361 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:23.689463 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:23.689415 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" event={"ID":"08de0f2d-5d41-4ed7-a4e3-2413b16271a5","Type":"ContainerStarted","Data":"e1406d2a61a7eaec64c94a4351f8fff2046d99f62c7abaddc92cff846dc8b30d"} Aug 04 00:35:23.704955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:23.704906 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" podStartSLOduration=1.704891648 podStartE2EDuration="1.704891648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:23.70343093 +0000 UTC m=+1961.874464238" watchObservedRunningTime="2026-08-04 00:35:23.704891648 +0000 UTC m=+1961.875924954" Aug 04 00:35:24.696303 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:24.696258 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" event={"ID":"258ed20e-98b8-4e2c-b6fe-549517813f79","Type":"ContainerStarted","Data":"934d81ab94bdd5591fdcab051d4bde465b4aa3bd7abbf4f4606b4c625760095b"} Aug 04 00:35:24.696726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:24.696592 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:24.711616 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:24.711572 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" podStartSLOduration=1.986362849 podStartE2EDuration="3.711557175s" totalImagesPullingTime="1.725194326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:24.710496428 +0000 UTC m=+1962.881529765" watchObservedRunningTime="2026-08-04 00:35:24.711557175 +0000 UTC m=+1962.882590481" Aug 04 00:35:30.708138 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:30.708095 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:34.698390 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:34.698247 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:52.609121 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.609074 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v"] Aug 04 00:35:52.609584 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.609435 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" podUID="258ed20e-98b8-4e2c-b6fe-549517813f79" containerName="maas-api" containerID="cri-o://934d81ab94bdd5591fdcab051d4bde465b4aa3bd7abbf4f4606b4c625760095b" gracePeriod=30 Aug 04 00:35:52.661026 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.660994 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh"] Aug 04 00:35:52.661262 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.661236 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" podUID="08de0f2d-5d41-4ed7-a4e3-2413b16271a5" containerName="payload-processing" containerID="cri-o://2063df50a69c39b6c7b47acdab9a4cb3c75572876d1c2e14220ee7dc1282d40c" gracePeriod=30 Aug 04 00:35:52.855872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.855837 2571 generic.go:356] "Generic (PLEG): container finished" podID="08de0f2d-5d41-4ed7-a4e3-2413b16271a5" containerID="2063df50a69c39b6c7b47acdab9a4cb3c75572876d1c2e14220ee7dc1282d40c" exitCode=0 Aug 04 00:35:52.856249 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.856040 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" event={"ID":"08de0f2d-5d41-4ed7-a4e3-2413b16271a5","Type":"ContainerDied","Data":"2063df50a69c39b6c7b47acdab9a4cb3c75572876d1c2e14220ee7dc1282d40c"} Aug 04 00:35:52.858826 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.858796 2571 generic.go:356] "Generic (PLEG): container finished" podID="258ed20e-98b8-4e2c-b6fe-549517813f79" containerID="934d81ab94bdd5591fdcab051d4bde465b4aa3bd7abbf4f4606b4c625760095b" exitCode=0 Aug 04 00:35:52.859208 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.858836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" event={"ID":"258ed20e-98b8-4e2c-b6fe-549517813f79","Type":"ContainerDied","Data":"934d81ab94bdd5591fdcab051d4bde465b4aa3bd7abbf4f4606b4c625760095b"} Aug 04 00:35:52.909650 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.909626 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:52.937874 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:52.937852 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:53.096618 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.096586 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258ed20e-98b8-4e2c-b6fe-549517813f79-openshift-service-ca\") pod \"258ed20e-98b8-4e2c-b6fe-549517813f79\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " Aug 04 00:35:53.096618 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.096624 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-plugins-config-volume\") pod \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " Aug 04 00:35:53.096948 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.096667 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258ed20e-98b8-4e2c-b6fe-549517813f79-maas-api-tls\") pod \"258ed20e-98b8-4e2c-b6fe-549517813f79\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " Aug 04 00:35:53.096948 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.096734 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-kube-api-access-wdbph\") pod \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\" (UID: \"08de0f2d-5d41-4ed7-a4e3-2413b16271a5\") " Aug 04 00:35:53.096948 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.096809 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdd9x\" (UniqueName: \"kubernetes.io/projected/258ed20e-98b8-4e2c-b6fe-549517813f79-kube-api-access-sdd9x\") pod \"258ed20e-98b8-4e2c-b6fe-549517813f79\" (UID: \"258ed20e-98b8-4e2c-b6fe-549517813f79\") " Aug 04 00:35:53.097181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.097012 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258ed20e-98b8-4e2c-b6fe-549517813f79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "258ed20e-98b8-4e2c-b6fe-549517813f79" (UID: "258ed20e-98b8-4e2c-b6fe-549517813f79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:35:53.097181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.097055 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08de0f2d-5d41-4ed7-a4e3-2413b16271a5" (UID: "08de0f2d-5d41-4ed7-a4e3-2413b16271a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:35:53.097181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.097078 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/258ed20e-98b8-4e2c-b6fe-549517813f79-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.099020 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.098997 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258ed20e-98b8-4e2c-b6fe-549517813f79-kube-api-access-sdd9x" (OuterVolumeSpecName: "kube-api-access-sdd9x") pod "258ed20e-98b8-4e2c-b6fe-549517813f79" (UID: "258ed20e-98b8-4e2c-b6fe-549517813f79"). InnerVolumeSpecName "kube-api-access-sdd9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:35:53.099130 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.099018 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258ed20e-98b8-4e2c-b6fe-549517813f79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "258ed20e-98b8-4e2c-b6fe-549517813f79" (UID: "258ed20e-98b8-4e2c-b6fe-549517813f79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:35:53.099130 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.099024 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-kube-api-access-wdbph" (OuterVolumeSpecName: "kube-api-access-wdbph") pod "08de0f2d-5d41-4ed7-a4e3-2413b16271a5" (UID: "08de0f2d-5d41-4ed7-a4e3-2413b16271a5"). InnerVolumeSpecName "kube-api-access-wdbph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:35:53.198394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.198306 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.198394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.198336 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/258ed20e-98b8-4e2c-b6fe-549517813f79-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.198394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.198346 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbph\" (UniqueName: \"kubernetes.io/projected/08de0f2d-5d41-4ed7-a4e3-2413b16271a5-kube-api-access-wdbph\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.198394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.198358 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdd9x\" (UniqueName: \"kubernetes.io/projected/258ed20e-98b8-4e2c-b6fe-549517813f79-kube-api-access-sdd9x\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:35:53.866021 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.865987 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" Aug 04 00:35:53.866021 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.866003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh" event={"ID":"08de0f2d-5d41-4ed7-a4e3-2413b16271a5","Type":"ContainerDied","Data":"e1406d2a61a7eaec64c94a4351f8fff2046d99f62c7abaddc92cff846dc8b30d"} Aug 04 00:35:53.866607 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.866096 2571 scope.go:117] "RemoveContainer" containerID="2063df50a69c39b6c7b47acdab9a4cb3c75572876d1c2e14220ee7dc1282d40c" Aug 04 00:35:53.867654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.867505 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" event={"ID":"258ed20e-98b8-4e2c-b6fe-549517813f79","Type":"ContainerDied","Data":"60cca6f18731639c45cfb43ee72689a34961fa333924422dbc6d19407bcac7bd"} Aug 04 00:35:53.867654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.867563 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v" Aug 04 00:35:53.875447 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.875416 2571 scope.go:117] "RemoveContainer" containerID="934d81ab94bdd5591fdcab051d4bde465b4aa3bd7abbf4f4606b4c625760095b" Aug 04 00:35:53.895878 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.895855 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh"] Aug 04 00:35:53.897919 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.897894 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-418d3ab6-d45ccd7d-67brh"] Aug 04 00:35:53.912084 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.912062 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v"] Aug 04 00:35:53.916729 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:53.916708 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-418d3ab6-7556f9f67-dqp2v"] Aug 04 00:35:54.377214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:54.377180 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08de0f2d-5d41-4ed7-a4e3-2413b16271a5" path="/var/lib/kubelet/pods/08de0f2d-5d41-4ed7-a4e3-2413b16271a5/volumes" Aug 04 00:35:54.377669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:54.377653 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="258ed20e-98b8-4e2c-b6fe-549517813f79" path="/var/lib/kubelet/pods/258ed20e-98b8-4e2c-b6fe-549517813f79/volumes" Aug 04 00:35:56.246874 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.246844 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv"] Aug 04 00:35:56.247277 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.247218 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08de0f2d-5d41-4ed7-a4e3-2413b16271a5" containerName="payload-processing" Aug 04 00:35:56.247277 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.247230 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="08de0f2d-5d41-4ed7-a4e3-2413b16271a5" containerName="payload-processing" Aug 04 00:35:56.247277 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.247243 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258ed20e-98b8-4e2c-b6fe-549517813f79" containerName="maas-api" Aug 04 00:35:56.247277 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.247248 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="258ed20e-98b8-4e2c-b6fe-549517813f79" containerName="maas-api" Aug 04 00:35:56.247409 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.247314 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="08de0f2d-5d41-4ed7-a4e3-2413b16271a5" containerName="payload-processing" Aug 04 00:35:56.247409 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.247323 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="258ed20e-98b8-4e2c-b6fe-549517813f79" containerName="maas-api" Aug 04 00:35:56.251826 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.251810 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.254403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.254383 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ae02d212-serving-cert\"" Aug 04 00:35:56.275233 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.275209 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv"] Aug 04 00:35:56.324120 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.324066 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee995939-b49d-4142-90ba-c13f48cbe593-openshift-service-ca\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.324120 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.324116 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj46x\" (UniqueName: \"kubernetes.io/projected/ee995939-b49d-4142-90ba-c13f48cbe593-kube-api-access-gj46x\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.324335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.324190 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.425339 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.425304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.425575 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.425420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee995939-b49d-4142-90ba-c13f48cbe593-openshift-service-ca\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.425575 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.425456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj46x\" (UniqueName: \"kubernetes.io/projected/ee995939-b49d-4142-90ba-c13f48cbe593-kube-api-access-gj46x\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.425575 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:35:56.425457 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-serving-cert: secret "maas-api-e2e-migrate-ae02d212-serving-cert" not found Aug 04 00:35:56.425575 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:35:56.425572 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls podName:ee995939-b49d-4142-90ba-c13f48cbe593 nodeName:}" failed. No retries permitted until 2026-08-04 00:35:56.92555164 +0000 UTC m=+1995.096584938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls") pod "maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" (UID: "ee995939-b49d-4142-90ba-c13f48cbe593") : secret "maas-api-e2e-migrate-ae02d212-serving-cert" not found Aug 04 00:35:56.426064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.426043 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee995939-b49d-4142-90ba-c13f48cbe593-openshift-service-ca\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.433266 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.433245 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj46x\" (UniqueName: \"kubernetes.io/projected/ee995939-b49d-4142-90ba-c13f48cbe593-kube-api-access-gj46x\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.896130 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.896083 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh"] Aug 04 00:35:56.899927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.899906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:56.902159 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.902134 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ae02d212\"" Aug 04 00:35:56.902263 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.902178 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ae02d212-dockercfg-x5gk2\"" Aug 04 00:35:56.907523 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.907498 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh"] Aug 04 00:35:56.927719 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.927690 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g"] Aug 04 00:35:56.930594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.930576 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957j4\" (UniqueName: \"kubernetes.io/projected/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-kube-api-access-957j4\") pod \"payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:56.930678 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.930640 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.930678 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.930658 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:56.931221 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.931204 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:56.932968 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.932947 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls\") pod \"maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:56.942534 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:56.942511 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g"] Aug 04 00:35:57.031409 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.031359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:57.031755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.031461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvs9\" (UniqueName: \"kubernetes.io/projected/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-kube-api-access-kbvs9\") pod \"payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.031755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.031523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957j4\" (UniqueName: \"kubernetes.io/projected/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-kube-api-access-957j4\") pod \"payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:57.031755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.031624 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.032251 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.032229 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:57.039314 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.039282 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-957j4\" (UniqueName: \"kubernetes.io/projected/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-kube-api-access-957j4\") pod \"payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:57.132940 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.132896 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.133143 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.133004 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvs9\" (UniqueName: \"kubernetes.io/projected/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-kube-api-access-kbvs9\") pod \"payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.133742 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.133720 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.140841 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.140815 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvs9\" (UniqueName: \"kubernetes.io/projected/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-kube-api-access-kbvs9\") pod \"payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.161847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.161777 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:57.212058 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.211786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:57.249690 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.249636 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.311399 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.311360 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv"] Aug 04 00:35:57.313763 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:35:57.313730 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee995939_b49d_4142_90ba_c13f48cbe593.slice/crio-e50f57289d725e613f0282e49ffcd3e6f9c96cfffcbaffce2b9f42f214ff1a60 WatchSource:0}: Error finding container e50f57289d725e613f0282e49ffcd3e6f9c96cfffcbaffce2b9f42f214ff1a60: Status 404 returned error can't find the container with id e50f57289d725e613f0282e49ffcd3e6f9c96cfffcbaffce2b9f42f214ff1a60 Aug 04 00:35:57.377922 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.377894 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh"] Aug 04 00:35:57.380157 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:35:57.380120 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e6ebce_d4eb_4a40_86d3_a117b32f56d3.slice/crio-8304263af14f9642fa3991917cbf4ce10869f28505911e8da5cff436365adbe7 WatchSource:0}: Error finding container 8304263af14f9642fa3991917cbf4ce10869f28505911e8da5cff436365adbe7: Status 404 returned error can't find the container with id 8304263af14f9642fa3991917cbf4ce10869f28505911e8da5cff436365adbe7 Aug 04 00:35:57.411905 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.411835 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g"] Aug 04 00:35:57.416278 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:35:57.416242 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a6e38d_0f3d_4bb4_803f_9d0cb70542b0.slice/crio-3db2d342452a7f655186b7a5c972eb9f3cd81829c5b9b0de76e0ac9a31540d53 WatchSource:0}: Error finding container 3db2d342452a7f655186b7a5c972eb9f3cd81829c5b9b0de76e0ac9a31540d53: Status 404 returned error can't find the container with id 3db2d342452a7f655186b7a5c972eb9f3cd81829c5b9b0de76e0ac9a31540d53 Aug 04 00:35:57.893568 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.893521 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" event={"ID":"ee995939-b49d-4142-90ba-c13f48cbe593","Type":"ContainerStarted","Data":"a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557"} Aug 04 00:35:57.893896 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.893868 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:35:57.894026 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.893909 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" event={"ID":"ee995939-b49d-4142-90ba-c13f48cbe593","Type":"ContainerStarted","Data":"e50f57289d725e613f0282e49ffcd3e6f9c96cfffcbaffce2b9f42f214ff1a60"} Aug 04 00:35:57.895213 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.895185 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" event={"ID":"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3","Type":"ContainerStarted","Data":"80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0"} Aug 04 00:35:57.895371 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.895235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" event={"ID":"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3","Type":"ContainerStarted","Data":"8304263af14f9642fa3991917cbf4ce10869f28505911e8da5cff436365adbe7"} Aug 04 00:35:57.895371 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.895262 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:35:57.896835 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.896813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" event={"ID":"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0","Type":"ContainerStarted","Data":"615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd"} Aug 04 00:35:57.896921 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.896849 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" event={"ID":"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0","Type":"ContainerStarted","Data":"3db2d342452a7f655186b7a5c972eb9f3cd81829c5b9b0de76e0ac9a31540d53"} Aug 04 00:35:57.896921 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.896883 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:35:57.910253 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.910212 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" podStartSLOduration=1.590359979 podStartE2EDuration="1.910197806s" totalImagesPullingTime="319.837827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:57.907832708 +0000 UTC m=+1996.078866014" watchObservedRunningTime="2026-08-04 00:35:57.910197806 +0000 UTC m=+1996.081231113" Aug 04 00:35:57.922839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.922764 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" podStartSLOduration=1.9227442080000001 podStartE2EDuration="1.922744208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:57.92123085 +0000 UTC m=+1996.092264158" watchObservedRunningTime="2026-08-04 00:35:57.922744208 +0000 UTC m=+1996.093777515" Aug 04 00:35:57.935547 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:35:57.935497 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" podStartSLOduration=1.935479098 podStartE2EDuration="1.935479098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:35:57.934704057 +0000 UTC m=+1996.105737366" watchObservedRunningTime="2026-08-04 00:35:57.935479098 +0000 UTC m=+1996.106512418" Aug 04 00:36:03.908438 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:03.908407 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:36:08.905083 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:08.905055 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:36:08.905606 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:08.905343 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:36:27.487515 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.487470 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv"] Aug 04 00:36:27.487970 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.487722 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" podUID="ee995939-b49d-4142-90ba-c13f48cbe593" containerName="maas-api" containerID="cri-o://a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557" gracePeriod=30 Aug 04 00:36:27.541421 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.541377 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g"] Aug 04 00:36:27.541669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.541640 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" podUID="33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" containerName="payload-processing" containerID="cri-o://615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd" gracePeriod=30 Aug 04 00:36:27.548929 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.548897 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh"] Aug 04 00:36:27.549350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.549152 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" podUID="e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" containerName="payload-pre-processing" containerID="cri-o://80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0" gracePeriod=30 Aug 04 00:36:27.824338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.824309 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:36:27.844490 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.844464 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:36:27.851858 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.851832 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:36:27.917128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.916576 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj46x\" (UniqueName: \"kubernetes.io/projected/ee995939-b49d-4142-90ba-c13f48cbe593-kube-api-access-gj46x\") pod \"ee995939-b49d-4142-90ba-c13f48cbe593\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " Aug 04 00:36:27.917128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.916643 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee995939-b49d-4142-90ba-c13f48cbe593-openshift-service-ca\") pod \"ee995939-b49d-4142-90ba-c13f48cbe593\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " Aug 04 00:36:27.917128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.916682 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls\") pod \"ee995939-b49d-4142-90ba-c13f48cbe593\" (UID: \"ee995939-b49d-4142-90ba-c13f48cbe593\") " Aug 04 00:36:27.917128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.916755 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-plugins-config-volume\") pod \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " Aug 04 00:36:27.917128 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.916796 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbvs9\" (UniqueName: \"kubernetes.io/projected/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-kube-api-access-kbvs9\") pod \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\" (UID: \"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0\") " Aug 04 00:36:27.917529 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.917338 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" (UID: "33a6e38d-0f3d-4bb4-803f-9d0cb70542b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:36:27.918189 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.917812 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee995939-b49d-4142-90ba-c13f48cbe593-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee995939-b49d-4142-90ba-c13f48cbe593" (UID: "ee995939-b49d-4142-90ba-c13f48cbe593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:36:27.919644 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.919607 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-kube-api-access-kbvs9" (OuterVolumeSpecName: "kube-api-access-kbvs9") pod "33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" (UID: "33a6e38d-0f3d-4bb4-803f-9d0cb70542b0"). InnerVolumeSpecName "kube-api-access-kbvs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:36:27.919644 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.919642 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee995939-b49d-4142-90ba-c13f48cbe593-kube-api-access-gj46x" (OuterVolumeSpecName: "kube-api-access-gj46x") pod "ee995939-b49d-4142-90ba-c13f48cbe593" (UID: "ee995939-b49d-4142-90ba-c13f48cbe593"). InnerVolumeSpecName "kube-api-access-gj46x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:36:27.920420 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:27.920403 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee995939-b49d-4142-90ba-c13f48cbe593" (UID: "ee995939-b49d-4142-90ba-c13f48cbe593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:36:28.018068 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018036 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-957j4\" (UniqueName: \"kubernetes.io/projected/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-kube-api-access-957j4\") pod \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " Aug 04 00:36:28.018257 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018080 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-plugins-config-volume\") pod \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\" (UID: \"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3\") " Aug 04 00:36:28.018257 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018242 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.018257 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018254 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbvs9\" (UniqueName: \"kubernetes.io/projected/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0-kube-api-access-kbvs9\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.018412 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018263 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj46x\" (UniqueName: \"kubernetes.io/projected/ee995939-b49d-4142-90ba-c13f48cbe593-kube-api-access-gj46x\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.018412 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018271 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee995939-b49d-4142-90ba-c13f48cbe593-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.018412 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018280 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee995939-b49d-4142-90ba-c13f48cbe593-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.018557 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.018521 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" (UID: "e0e6ebce-d4eb-4a40-86d3-a117b32f56d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:36:28.020193 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.020171 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-kube-api-access-957j4" (OuterVolumeSpecName: "kube-api-access-957j4") pod "e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" (UID: "e0e6ebce-d4eb-4a40-86d3-a117b32f56d3"). InnerVolumeSpecName "kube-api-access-957j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:36:28.059636 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.059598 2571 generic.go:356] "Generic (PLEG): container finished" podID="ee995939-b49d-4142-90ba-c13f48cbe593" containerID="a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557" exitCode=0 Aug 04 00:36:28.059821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.059650 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" Aug 04 00:36:28.059821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.059689 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" event={"ID":"ee995939-b49d-4142-90ba-c13f48cbe593","Type":"ContainerDied","Data":"a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557"} Aug 04 00:36:28.059821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.059742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv" event={"ID":"ee995939-b49d-4142-90ba-c13f48cbe593","Type":"ContainerDied","Data":"e50f57289d725e613f0282e49ffcd3e6f9c96cfffcbaffce2b9f42f214ff1a60"} Aug 04 00:36:28.059821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.059766 2571 scope.go:117] "RemoveContainer" containerID="a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557" Aug 04 00:36:28.061202 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.061183 2571 generic.go:356] "Generic (PLEG): container finished" podID="e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" containerID="80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0" exitCode=0 Aug 04 00:36:28.061303 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.061225 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" Aug 04 00:36:28.061303 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.061264 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" event={"ID":"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3","Type":"ContainerDied","Data":"80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0"} Aug 04 00:36:28.061430 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.061327 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh" event={"ID":"e0e6ebce-d4eb-4a40-86d3-a117b32f56d3","Type":"ContainerDied","Data":"8304263af14f9642fa3991917cbf4ce10869f28505911e8da5cff436365adbe7"} Aug 04 00:36:28.062900 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.062867 2571 generic.go:356] "Generic (PLEG): container finished" podID="33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" containerID="615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd" exitCode=0 Aug 04 00:36:28.063022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.062923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" event={"ID":"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0","Type":"ContainerDied","Data":"615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd"} Aug 04 00:36:28.063022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.062938 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" Aug 04 00:36:28.063022 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.062964 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g" event={"ID":"33a6e38d-0f3d-4bb4-803f-9d0cb70542b0","Type":"ContainerDied","Data":"3db2d342452a7f655186b7a5c972eb9f3cd81829c5b9b0de76e0ac9a31540d53"} Aug 04 00:36:28.071408 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.071308 2571 scope.go:117] "RemoveContainer" containerID="a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557" Aug 04 00:36:28.071765 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:36:28.071731 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557\": container with ID starting with a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557 not found: ID does not exist" containerID="a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557" Aug 04 00:36:28.071852 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.071776 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557"} err="failed to get container status \"a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557\": rpc error: code = NotFound desc = could not find container \"a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557\": container with ID starting with a55bfabac130320bcbb349e2aa1057931b2e0a781a6329d2c251ffb0828e6557 not found: ID does not exist" Aug 04 00:36:28.071852 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.071800 2571 scope.go:117] "RemoveContainer" containerID="80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0" Aug 04 00:36:28.080298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.080278 2571 scope.go:117] "RemoveContainer" containerID="80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0" Aug 04 00:36:28.080563 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:36:28.080548 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0\": container with ID starting with 80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0 not found: ID does not exist" containerID="80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0" Aug 04 00:36:28.080620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.080569 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0"} err="failed to get container status \"80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0\": rpc error: code = NotFound desc = could not find container \"80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0\": container with ID starting with 80cb6ee07a61b8153549fa54f447312ca5588b0f2905c2c0bcb7c450daa735c0 not found: ID does not exist" Aug 04 00:36:28.080620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.080584 2571 scope.go:117] "RemoveContainer" containerID="615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd" Aug 04 00:36:28.085079 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.085054 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv"] Aug 04 00:36:28.087442 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.087420 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ae02d212-56bd977c78-w4wgv"] Aug 04 00:36:28.089163 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.089142 2571 scope.go:117] "RemoveContainer" containerID="615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd" Aug 04 00:36:28.089424 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:36:28.089402 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd\": container with ID starting with 615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd not found: ID does not exist" containerID="615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd" Aug 04 00:36:28.089497 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.089434 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd"} err="failed to get container status \"615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd\": rpc error: code = NotFound desc = could not find container \"615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd\": container with ID starting with 615b83c79dc4de25d10fa37c8668e4bbbd82c91676a9f81e6d47779e1b9320dd not found: ID does not exist" Aug 04 00:36:28.095974 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.095945 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh"] Aug 04 00:36:28.101261 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.101232 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ae02d212-7b65f9cf4d-q6dfh"] Aug 04 00:36:28.111978 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.111953 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g"] Aug 04 00:36:28.115737 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.115714 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ae02d212-8575745dfc-w7k5g"] Aug 04 00:36:28.119490 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.119467 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957j4\" (UniqueName: \"kubernetes.io/projected/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-kube-api-access-957j4\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.119608 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.119496 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:36:28.377238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.377162 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" path="/var/lib/kubelet/pods/33a6e38d-0f3d-4bb4-803f-9d0cb70542b0/volumes" Aug 04 00:36:28.377555 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.377540 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" path="/var/lib/kubelet/pods/e0e6ebce-d4eb-4a40-86d3-a117b32f56d3/volumes" Aug 04 00:36:28.377889 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:28.377878 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee995939-b49d-4142-90ba-c13f48cbe593" path="/var/lib/kubelet/pods/ee995939-b49d-4142-90ba-c13f48cbe593/volumes" Aug 04 00:36:32.291002 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.290967 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2"] Aug 04 00:36:32.291528 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291504 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" containerName="payload-pre-processing" Aug 04 00:36:32.291593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291531 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" containerName="payload-pre-processing" Aug 04 00:36:32.291593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291547 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee995939-b49d-4142-90ba-c13f48cbe593" containerName="maas-api" Aug 04 00:36:32.291593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291553 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee995939-b49d-4142-90ba-c13f48cbe593" containerName="maas-api" Aug 04 00:36:32.291593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291563 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" containerName="payload-processing" Aug 04 00:36:32.291593 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291568 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" containerName="payload-processing" Aug 04 00:36:32.291764 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291651 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e6ebce-d4eb-4a40-86d3-a117b32f56d3" containerName="payload-pre-processing" Aug 04 00:36:32.291764 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291661 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a6e38d-0f3d-4bb4-803f-9d0cb70542b0" containerName="payload-processing" Aug 04 00:36:32.291764 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.291673 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee995939-b49d-4142-90ba-c13f48cbe593" containerName="maas-api" Aug 04 00:36:32.296046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.296031 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.298799 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.298778 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-62613146-serving-cert\"" Aug 04 00:36:32.307164 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.307136 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2"] Aug 04 00:36:32.461981 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.461948 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbq6\" (UniqueName: \"kubernetes.io/projected/f186329b-faf7-44ec-8b90-bdb5636fb3ec-kube-api-access-slbq6\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.462193 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.462014 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186329b-faf7-44ec-8b90-bdb5636fb3ec-openshift-service-ca\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.462193 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.462056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.563416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.563330 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbq6\" (UniqueName: \"kubernetes.io/projected/f186329b-faf7-44ec-8b90-bdb5636fb3ec-kube-api-access-slbq6\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.563416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.563389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186329b-faf7-44ec-8b90-bdb5636fb3ec-openshift-service-ca\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.563629 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.563429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.563629 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:36:32.563524 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-62613146-serving-cert: secret "maas-api-e2e-ait-62613146-serving-cert" not found Aug 04 00:36:32.563629 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:36:32.563579 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls podName:f186329b-faf7-44ec-8b90-bdb5636fb3ec nodeName:}" failed. No retries permitted until 2026-08-04 00:36:33.063562708 +0000 UTC m=+2031.234595997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls") pod "maas-api-e2e-ait-62613146-846d8fbf67-c42z2" (UID: "f186329b-faf7-44ec-8b90-bdb5636fb3ec") : secret "maas-api-e2e-ait-62613146-serving-cert" not found Aug 04 00:36:32.564092 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.564072 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186329b-faf7-44ec-8b90-bdb5636fb3ec-openshift-service-ca\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.573289 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.573269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbq6\" (UniqueName: \"kubernetes.io/projected/f186329b-faf7-44ec-8b90-bdb5636fb3ec-kube-api-access-slbq6\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:32.927938 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.927861 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf"] Aug 04 00:36:32.931685 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.931664 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:32.934037 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.934017 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-62613146-dockercfg-vg87t\"" Aug 04 00:36:32.934512 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.934498 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-62613146\"" Aug 04 00:36:32.944439 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.944410 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf"] Aug 04 00:36:32.969373 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.969345 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp"] Aug 04 00:36:32.972667 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.972649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:32.990646 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:32.990619 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp"] Aug 04 00:36:33.068288 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.068247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.068472 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.068314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dc683-353b-4f88-82af-a13ba3ede235-plugins-config-volume\") pod \"payload-processing-e2e-ait-62613146-98796fb8b-rvzhp\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.068472 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.068370 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkb2\" (UniqueName: \"kubernetes.io/projected/7b5dc683-353b-4f88-82af-a13ba3ede235-kube-api-access-fzkb2\") pod \"payload-processing-e2e-ait-62613146-98796fb8b-rvzhp\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.068472 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.068430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:33.068472 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.068466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgwt\" (UniqueName: \"kubernetes.io/projected/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-kube-api-access-qjgwt\") pod \"payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.070896 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.070870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls\") pod \"maas-api-e2e-ait-62613146-846d8fbf67-c42z2\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:33.169718 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.169682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkb2\" (UniqueName: \"kubernetes.io/projected/7b5dc683-353b-4f88-82af-a13ba3ede235-kube-api-access-fzkb2\") pod \"payload-processing-e2e-ait-62613146-98796fb8b-rvzhp\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.169891 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.169749 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgwt\" (UniqueName: \"kubernetes.io/projected/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-kube-api-access-qjgwt\") pod \"payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.169891 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.169782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.169891 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.169816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dc683-353b-4f88-82af-a13ba3ede235-plugins-config-volume\") pod \"payload-processing-e2e-ait-62613146-98796fb8b-rvzhp\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.170566 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.170545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dc683-353b-4f88-82af-a13ba3ede235-plugins-config-volume\") pod \"payload-processing-e2e-ait-62613146-98796fb8b-rvzhp\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.170566 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.170558 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.178175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.178068 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkb2\" (UniqueName: \"kubernetes.io/projected/7b5dc683-353b-4f88-82af-a13ba3ede235-kube-api-access-fzkb2\") pod \"payload-processing-e2e-ait-62613146-98796fb8b-rvzhp\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.178175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.178118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgwt\" (UniqueName: \"kubernetes.io/projected/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-kube-api-access-qjgwt\") pod \"payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.207143 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.207086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:33.241830 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.241799 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:33.282769 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.282739 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:33.368061 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.368010 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2"] Aug 04 00:36:33.397083 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.397053 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf"] Aug 04 00:36:33.401037 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:36:33.401007 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e94276b_27b2_4201_98e6_f4ffd3e6bc68.slice/crio-075ee39c9cf835da824249eca24c36f54e12de0e32ff81fa4e03dda80c6a5229 WatchSource:0}: Error finding container 075ee39c9cf835da824249eca24c36f54e12de0e32ff81fa4e03dda80c6a5229: Status 404 returned error can't find the container with id 075ee39c9cf835da824249eca24c36f54e12de0e32ff81fa4e03dda80c6a5229 Aug 04 00:36:33.455802 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:33.455773 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp"] Aug 04 00:36:33.457720 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:36:33.457691 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5dc683_353b_4f88_82af_a13ba3ede235.slice/crio-63c6c3d8ff51f12af1aabb8e9dc6a69b55031e8214392e95c1c6753971711fdf WatchSource:0}: Error finding container 63c6c3d8ff51f12af1aabb8e9dc6a69b55031e8214392e95c1c6753971711fdf: Status 404 returned error can't find the container with id 63c6c3d8ff51f12af1aabb8e9dc6a69b55031e8214392e95c1c6753971711fdf Aug 04 00:36:34.097786 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.097740 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" event={"ID":"7b5dc683-353b-4f88-82af-a13ba3ede235","Type":"ContainerStarted","Data":"8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0"} Aug 04 00:36:34.097983 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.097858 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:36:34.097983 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.097892 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" event={"ID":"7b5dc683-353b-4f88-82af-a13ba3ede235","Type":"ContainerStarted","Data":"63c6c3d8ff51f12af1aabb8e9dc6a69b55031e8214392e95c1c6753971711fdf"} Aug 04 00:36:34.099378 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.099345 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" event={"ID":"f186329b-faf7-44ec-8b90-bdb5636fb3ec","Type":"ContainerStarted","Data":"80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7"} Aug 04 00:36:34.099504 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.099396 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" event={"ID":"f186329b-faf7-44ec-8b90-bdb5636fb3ec","Type":"ContainerStarted","Data":"5f56c375c300a14e05aa10cbc6fde4aa414b858f4b02a5bd563283cbdae947d8"} Aug 04 00:36:34.099504 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.099461 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:34.101133 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.101097 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" event={"ID":"9e94276b-27b2-4201-98e6-f4ffd3e6bc68","Type":"ContainerStarted","Data":"0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b"} Aug 04 00:36:34.101218 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.101145 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" event={"ID":"9e94276b-27b2-4201-98e6-f4ffd3e6bc68","Type":"ContainerStarted","Data":"075ee39c9cf835da824249eca24c36f54e12de0e32ff81fa4e03dda80c6a5229"} Aug 04 00:36:34.101218 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.101179 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:34.113337 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.113291 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" podStartSLOduration=2.113276681 podStartE2EDuration="2.113276681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:36:34.11177506 +0000 UTC m=+2032.282808367" watchObservedRunningTime="2026-08-04 00:36:34.113276681 +0000 UTC m=+2032.284309988" Aug 04 00:36:34.127914 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.127829 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" podStartSLOduration=1.631012808 podStartE2EDuration="2.127816967s" totalImagesPullingTime="496.804159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:36:34.126406669 +0000 UTC m=+2032.297439997" watchObservedRunningTime="2026-08-04 00:36:34.127816967 +0000 UTC m=+2032.298850328" Aug 04 00:36:34.140877 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:34.140836 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" podStartSLOduration=2.140824192 podStartE2EDuration="2.140824192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:36:34.139281887 +0000 UTC m=+2032.310315195" watchObservedRunningTime="2026-08-04 00:36:34.140824192 +0000 UTC m=+2032.311857498" Aug 04 00:36:40.111979 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:40.111947 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:36:45.109051 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:45.109019 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:36:45.109521 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:36:45.109282 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:37:12.773671 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:12.773633 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2"] Aug 04 00:37:12.776674 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:12.773906 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" podUID="f186329b-faf7-44ec-8b90-bdb5636fb3ec" containerName="maas-api" containerID="cri-o://80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7" gracePeriod=30 Aug 04 00:37:12.830082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:12.830045 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp"] Aug 04 00:37:12.830343 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:12.830311 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" podUID="7b5dc683-353b-4f88-82af-a13ba3ede235" containerName="payload-processing" containerID="cri-o://8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0" gracePeriod=30 Aug 04 00:37:12.839354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:12.839320 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf"] Aug 04 00:37:12.839608 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:12.839555 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" podUID="9e94276b-27b2-4201-98e6-f4ffd3e6bc68" containerName="payload-pre-processing" containerID="cri-o://0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b" gracePeriod=30 Aug 04 00:37:13.096743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.096709 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:37:13.129832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.129802 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:37:13.134244 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.134220 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:37:13.213554 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213523 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-plugins-config-volume\") pod \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " Aug 04 00:37:13.213760 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213622 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slbq6\" (UniqueName: \"kubernetes.io/projected/f186329b-faf7-44ec-8b90-bdb5636fb3ec-kube-api-access-slbq6\") pod \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " Aug 04 00:37:13.213760 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213691 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls\") pod \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " Aug 04 00:37:13.213894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213779 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzkb2\" (UniqueName: \"kubernetes.io/projected/7b5dc683-353b-4f88-82af-a13ba3ede235-kube-api-access-fzkb2\") pod \"7b5dc683-353b-4f88-82af-a13ba3ede235\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " Aug 04 00:37:13.213894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213837 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186329b-faf7-44ec-8b90-bdb5636fb3ec-openshift-service-ca\") pod \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\" (UID: \"f186329b-faf7-44ec-8b90-bdb5636fb3ec\") " Aug 04 00:37:13.213894 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213862 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dc683-353b-4f88-82af-a13ba3ede235-plugins-config-volume\") pod \"7b5dc683-353b-4f88-82af-a13ba3ede235\" (UID: \"7b5dc683-353b-4f88-82af-a13ba3ede235\") " Aug 04 00:37:13.214274 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.213910 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjgwt\" (UniqueName: \"kubernetes.io/projected/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-kube-api-access-qjgwt\") pod \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\" (UID: \"9e94276b-27b2-4201-98e6-f4ffd3e6bc68\") " Aug 04 00:37:13.214274 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.214000 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e94276b-27b2-4201-98e6-f4ffd3e6bc68" (UID: "9e94276b-27b2-4201-98e6-f4ffd3e6bc68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:13.214412 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.214290 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f186329b-faf7-44ec-8b90-bdb5636fb3ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f186329b-faf7-44ec-8b90-bdb5636fb3ec" (UID: "f186329b-faf7-44ec-8b90-bdb5636fb3ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:13.214468 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.214430 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.214468 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.214451 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186329b-faf7-44ec-8b90-bdb5636fb3ec-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.214930 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.214897 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5dc683-353b-4f88-82af-a13ba3ede235-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b5dc683-353b-4f88-82af-a13ba3ede235" (UID: "7b5dc683-353b-4f88-82af-a13ba3ede235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:13.216061 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.216029 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f186329b-faf7-44ec-8b90-bdb5636fb3ec" (UID: "f186329b-faf7-44ec-8b90-bdb5636fb3ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:37:13.216061 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.216050 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f186329b-faf7-44ec-8b90-bdb5636fb3ec-kube-api-access-slbq6" (OuterVolumeSpecName: "kube-api-access-slbq6") pod "f186329b-faf7-44ec-8b90-bdb5636fb3ec" (UID: "f186329b-faf7-44ec-8b90-bdb5636fb3ec"). InnerVolumeSpecName "kube-api-access-slbq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:13.216295 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.216266 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5dc683-353b-4f88-82af-a13ba3ede235-kube-api-access-fzkb2" (OuterVolumeSpecName: "kube-api-access-fzkb2") pod "7b5dc683-353b-4f88-82af-a13ba3ede235" (UID: "7b5dc683-353b-4f88-82af-a13ba3ede235"). InnerVolumeSpecName "kube-api-access-fzkb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:13.216472 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.216444 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-kube-api-access-qjgwt" (OuterVolumeSpecName: "kube-api-access-qjgwt") pod "9e94276b-27b2-4201-98e6-f4ffd3e6bc68" (UID: "9e94276b-27b2-4201-98e6-f4ffd3e6bc68"). InnerVolumeSpecName "kube-api-access-qjgwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:13.314931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.314845 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgwt\" (UniqueName: \"kubernetes.io/projected/9e94276b-27b2-4201-98e6-f4ffd3e6bc68-kube-api-access-qjgwt\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.314931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.314872 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbq6\" (UniqueName: \"kubernetes.io/projected/f186329b-faf7-44ec-8b90-bdb5636fb3ec-kube-api-access-slbq6\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.314931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.314883 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f186329b-faf7-44ec-8b90-bdb5636fb3ec-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.314931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.314892 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzkb2\" (UniqueName: \"kubernetes.io/projected/7b5dc683-353b-4f88-82af-a13ba3ede235-kube-api-access-fzkb2\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.314931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.314901 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b5dc683-353b-4f88-82af-a13ba3ede235-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:13.319540 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.319510 2571 generic.go:356] "Generic (PLEG): container finished" podID="7b5dc683-353b-4f88-82af-a13ba3ede235" containerID="8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0" exitCode=0 Aug 04 00:37:13.319689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.319555 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" Aug 04 00:37:13.319689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.319600 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" event={"ID":"7b5dc683-353b-4f88-82af-a13ba3ede235","Type":"ContainerDied","Data":"8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0"} Aug 04 00:37:13.319689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.319661 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp" event={"ID":"7b5dc683-353b-4f88-82af-a13ba3ede235","Type":"ContainerDied","Data":"63c6c3d8ff51f12af1aabb8e9dc6a69b55031e8214392e95c1c6753971711fdf"} Aug 04 00:37:13.319868 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.319693 2571 scope.go:117] "RemoveContainer" containerID="8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0" Aug 04 00:37:13.320943 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.320923 2571 generic.go:356] "Generic (PLEG): container finished" podID="f186329b-faf7-44ec-8b90-bdb5636fb3ec" containerID="80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7" exitCode=0 Aug 04 00:37:13.321047 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.320974 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" Aug 04 00:37:13.321047 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.321010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" event={"ID":"f186329b-faf7-44ec-8b90-bdb5636fb3ec","Type":"ContainerDied","Data":"80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7"} Aug 04 00:37:13.321210 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.321065 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2" event={"ID":"f186329b-faf7-44ec-8b90-bdb5636fb3ec","Type":"ContainerDied","Data":"5f56c375c300a14e05aa10cbc6fde4aa414b858f4b02a5bd563283cbdae947d8"} Aug 04 00:37:13.322578 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.322560 2571 generic.go:356] "Generic (PLEG): container finished" podID="9e94276b-27b2-4201-98e6-f4ffd3e6bc68" containerID="0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b" exitCode=0 Aug 04 00:37:13.322706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.322593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" event={"ID":"9e94276b-27b2-4201-98e6-f4ffd3e6bc68","Type":"ContainerDied","Data":"0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b"} Aug 04 00:37:13.322706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.322613 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" Aug 04 00:37:13.322706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.322625 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf" event={"ID":"9e94276b-27b2-4201-98e6-f4ffd3e6bc68","Type":"ContainerDied","Data":"075ee39c9cf835da824249eca24c36f54e12de0e32ff81fa4e03dda80c6a5229"} Aug 04 00:37:13.329216 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.329194 2571 scope.go:117] "RemoveContainer" containerID="8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0" Aug 04 00:37:13.329552 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:37:13.329532 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0\": container with ID starting with 8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0 not found: ID does not exist" containerID="8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0" Aug 04 00:37:13.329635 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.329562 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0"} err="failed to get container status \"8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0\": rpc error: code = NotFound desc = could not find container \"8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0\": container with ID starting with 8854a8e2d35599c2cf046c0320227ad02372960f23f30b5deef239034f6360f0 not found: ID does not exist" Aug 04 00:37:13.329635 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.329585 2571 scope.go:117] "RemoveContainer" containerID="80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7" Aug 04 00:37:13.339377 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.339362 2571 scope.go:117] "RemoveContainer" containerID="80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7" Aug 04 00:37:13.339659 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:37:13.339642 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7\": container with ID starting with 80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7 not found: ID does not exist" containerID="80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7" Aug 04 00:37:13.339703 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.339666 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7"} err="failed to get container status \"80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7\": rpc error: code = NotFound desc = could not find container \"80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7\": container with ID starting with 80d390f18decdd131b514d4da90458ea87d2652e8a7d3701fb25239668e5fca7 not found: ID does not exist" Aug 04 00:37:13.339703 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.339684 2571 scope.go:117] "RemoveContainer" containerID="0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b" Aug 04 00:37:13.347654 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.347632 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf"] Aug 04 00:37:13.348253 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.348233 2571 scope.go:117] "RemoveContainer" containerID="0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b" Aug 04 00:37:13.348519 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:37:13.348501 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b\": container with ID starting with 0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b not found: ID does not exist" containerID="0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b" Aug 04 00:37:13.348576 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.348525 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b"} err="failed to get container status \"0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b\": rpc error: code = NotFound desc = could not find container \"0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b\": container with ID starting with 0db97e138f608e3ba2a5d1200e0012c5beee3519405ab478db6b8852c152b44b not found: ID does not exist" Aug 04 00:37:13.349847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.349827 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62613146-5bf894d7f6-2v6nf"] Aug 04 00:37:13.358618 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.358591 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp"] Aug 04 00:37:13.360653 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.360629 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62613146-98796fb8b-rvzhp"] Aug 04 00:37:13.368552 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.368521 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2"] Aug 04 00:37:13.372639 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:13.372617 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62613146-846d8fbf67-c42z2"] Aug 04 00:37:14.377415 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:14.377373 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5dc683-353b-4f88-82af-a13ba3ede235" path="/var/lib/kubelet/pods/7b5dc683-353b-4f88-82af-a13ba3ede235/volumes" Aug 04 00:37:14.377804 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:14.377744 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e94276b-27b2-4201-98e6-f4ffd3e6bc68" path="/var/lib/kubelet/pods/9e94276b-27b2-4201-98e6-f4ffd3e6bc68/volumes" Aug 04 00:37:14.378088 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:14.378075 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f186329b-faf7-44ec-8b90-bdb5636fb3ec" path="/var/lib/kubelet/pods/f186329b-faf7-44ec-8b90-bdb5636fb3ec/volumes" Aug 04 00:37:18.534187 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534149 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6"] Aug 04 00:37:18.534683 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534663 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5dc683-353b-4f88-82af-a13ba3ede235" containerName="payload-processing" Aug 04 00:37:18.534766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534685 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5dc683-353b-4f88-82af-a13ba3ede235" containerName="payload-processing" Aug 04 00:37:18.534766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534712 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f186329b-faf7-44ec-8b90-bdb5636fb3ec" containerName="maas-api" Aug 04 00:37:18.534766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534721 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f186329b-faf7-44ec-8b90-bdb5636fb3ec" containerName="maas-api" Aug 04 00:37:18.534766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534731 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e94276b-27b2-4201-98e6-f4ffd3e6bc68" containerName="payload-pre-processing" Aug 04 00:37:18.534766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534740 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e94276b-27b2-4201-98e6-f4ffd3e6bc68" containerName="payload-pre-processing" Aug 04 00:37:18.535025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534837 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f186329b-faf7-44ec-8b90-bdb5636fb3ec" containerName="maas-api" Aug 04 00:37:18.535025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534854 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e94276b-27b2-4201-98e6-f4ffd3e6bc68" containerName="payload-pre-processing" Aug 04 00:37:18.535025 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.534866 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5dc683-353b-4f88-82af-a13ba3ede235" containerName="payload-processing" Aug 04 00:37:18.537938 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.537912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.540454 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.540433 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fe5216a4-serving-cert\"" Aug 04 00:37:18.547352 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.547327 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6"] Aug 04 00:37:18.666787 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.666753 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/43d03ed0-2f57-4b7a-a60a-0cc98309107d-kube-api-access-k529k\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.667000 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.666796 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.667000 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.666820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d03ed0-2f57-4b7a-a60a-0cc98309107d-openshift-service-ca\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.767902 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.767869 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/43d03ed0-2f57-4b7a-a60a-0cc98309107d-kube-api-access-k529k\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.768052 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.767908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.768052 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.767931 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d03ed0-2f57-4b7a-a60a-0cc98309107d-openshift-service-ca\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.768052 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:37:18.768041 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-serving-cert: secret "maas-api-e2e-derive-fe5216a4-serving-cert" not found Aug 04 00:37:18.768229 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:37:18.768128 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls podName:43d03ed0-2f57-4b7a-a60a-0cc98309107d nodeName:}" failed. No retries permitted until 2026-08-04 00:37:19.268090476 +0000 UTC m=+2077.439123779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls") pod "maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" (UID: "43d03ed0-2f57-4b7a-a60a-0cc98309107d") : secret "maas-api-e2e-derive-fe5216a4-serving-cert" not found Aug 04 00:37:18.768696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.768676 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d03ed0-2f57-4b7a-a60a-0cc98309107d-openshift-service-ca\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:18.777828 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:18.777807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/43d03ed0-2f57-4b7a-a60a-0cc98309107d-kube-api-access-k529k\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:19.187719 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.187684 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5"] Aug 04 00:37:19.191292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.191274 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.193535 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.193508 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fe5216a4-dockercfg-9lhph\"" Aug 04 00:37:19.193708 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.193691 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fe5216a4\"" Aug 04 00:37:19.200949 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.200925 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5"] Aug 04 00:37:19.223514 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.223480 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7"] Aug 04 00:37:19.227562 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.227539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.240029 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.240002 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7"] Aug 04 00:37:19.273882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.273849 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a655be2d-8d01-4a67-bba3-3620d3e3a321-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.274046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.273909 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:19.274046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.273937 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdg5w\" (UniqueName: \"kubernetes.io/projected/a655be2d-8d01-4a67-bba3-3620d3e3a321-kube-api-access-fdg5w\") pod \"payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.276342 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.276323 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls\") pod \"maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:19.374533 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.374500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-plugins-config-volume\") pod \"payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.374533 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.374538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcklv\" (UniqueName: \"kubernetes.io/projected/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-kube-api-access-lcklv\") pod \"payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.374754 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.374600 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a655be2d-8d01-4a67-bba3-3620d3e3a321-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.374754 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.374725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg5w\" (UniqueName: \"kubernetes.io/projected/a655be2d-8d01-4a67-bba3-3620d3e3a321-kube-api-access-fdg5w\") pod \"payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.375240 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.375222 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a655be2d-8d01-4a67-bba3-3620d3e3a321-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.383366 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.383345 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdg5w\" (UniqueName: \"kubernetes.io/projected/a655be2d-8d01-4a67-bba3-3620d3e3a321-kube-api-access-fdg5w\") pod \"payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.449335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.449244 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:19.475896 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.475869 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-plugins-config-volume\") pod \"payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.476046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.475908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcklv\" (UniqueName: \"kubernetes.io/projected/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-kube-api-access-lcklv\") pod \"payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.476772 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.476746 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-plugins-config-volume\") pod \"payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.484522 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.484498 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcklv\" (UniqueName: \"kubernetes.io/projected/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-kube-api-access-lcklv\") pod \"payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.502407 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.502325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:19.538741 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.538643 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:19.606202 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.605899 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6"] Aug 04 00:37:19.608693 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:37:19.608646 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d03ed0_2f57_4b7a_a60a_0cc98309107d.slice/crio-560b4a124eb48c8b18c568bdcc12f67459c0d803784957dbe5352c6f3c503c94 WatchSource:0}: Error finding container 560b4a124eb48c8b18c568bdcc12f67459c0d803784957dbe5352c6f3c503c94: Status 404 returned error can't find the container with id 560b4a124eb48c8b18c568bdcc12f67459c0d803784957dbe5352c6f3c503c94 Aug 04 00:37:19.666158 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.666126 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5"] Aug 04 00:37:19.668388 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:37:19.668344 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda655be2d_8d01_4a67_bba3_3620d3e3a321.slice/crio-89d3331f1a00553d0ba4bbaa7842b2bdc7a031e0cb45228ebff92689e683e844 WatchSource:0}: Error finding container 89d3331f1a00553d0ba4bbaa7842b2bdc7a031e0cb45228ebff92689e683e844: Status 404 returned error can't find the container with id 89d3331f1a00553d0ba4bbaa7842b2bdc7a031e0cb45228ebff92689e683e844 Aug 04 00:37:19.711421 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:19.711399 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7"] Aug 04 00:37:19.717176 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:37:19.717150 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9fc144_5c2d_4d8e_91f2_62fc5241927b.slice/crio-3ce9920c1d9a29e02970090fb5e5f034380c5c8dd63fb73019ee2d2aa8cc28f1 WatchSource:0}: Error finding container 3ce9920c1d9a29e02970090fb5e5f034380c5c8dd63fb73019ee2d2aa8cc28f1: Status 404 returned error can't find the container with id 3ce9920c1d9a29e02970090fb5e5f034380c5c8dd63fb73019ee2d2aa8cc28f1 Aug 04 00:37:20.376931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.376903 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.376949 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" event={"ID":"4e9fc144-5c2d-4d8e-91f2-62fc5241927b","Type":"ContainerStarted","Data":"1a8685dd163792af484aa7f814f5e1179384b4d8f44f7c483853f29f6d67a27e"} Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.376969 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.376983 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" event={"ID":"4e9fc144-5c2d-4d8e-91f2-62fc5241927b","Type":"ContainerStarted","Data":"3ce9920c1d9a29e02970090fb5e5f034380c5c8dd63fb73019ee2d2aa8cc28f1"} Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.376997 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" event={"ID":"a655be2d-8d01-4a67-bba3-3620d3e3a321","Type":"ContainerStarted","Data":"d3704dc526675a5937eab204a6ce30d7f84f6bd43f8cd07df2e63e32d68e2958"} Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.377010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" event={"ID":"a655be2d-8d01-4a67-bba3-3620d3e3a321","Type":"ContainerStarted","Data":"89d3331f1a00553d0ba4bbaa7842b2bdc7a031e0cb45228ebff92689e683e844"} Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.377023 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.377037 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" event={"ID":"43d03ed0-2f57-4b7a-a60a-0cc98309107d","Type":"ContainerStarted","Data":"a61dbb6846674ff966b442630e12c58d90fa5383ca2837caf90c6506f11d6622"} Aug 04 00:37:20.377082 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.377049 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" event={"ID":"43d03ed0-2f57-4b7a-a60a-0cc98309107d","Type":"ContainerStarted","Data":"560b4a124eb48c8b18c568bdcc12f67459c0d803784957dbe5352c6f3c503c94"} Aug 04 00:37:20.388818 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.388762 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" podStartSLOduration=1.3887448359999999 podStartE2EDuration="1.388744836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:37:20.386475798 +0000 UTC m=+2078.557509105" watchObservedRunningTime="2026-08-04 00:37:20.388744836 +0000 UTC m=+2078.559778142" Aug 04 00:37:20.401514 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.401465 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" podStartSLOduration=1.401451848 podStartE2EDuration="1.401451848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:37:20.39968052 +0000 UTC m=+2078.570713825" watchObservedRunningTime="2026-08-04 00:37:20.401451848 +0000 UTC m=+2078.572485154" Aug 04 00:37:20.415362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:20.415322 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" podStartSLOduration=1.74736939 podStartE2EDuration="2.415308762s" totalImagesPullingTime="667.939372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:37:20.414624661 +0000 UTC m=+2078.585657980" watchObservedRunningTime="2026-08-04 00:37:20.415308762 +0000 UTC m=+2078.586342069" Aug 04 00:37:26.386200 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:26.386167 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:31.382735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:31.382701 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:31.383124 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:31.382881 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:49.237232 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.237197 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6"] Aug 04 00:37:49.237735 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.237467 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" podUID="43d03ed0-2f57-4b7a-a60a-0cc98309107d" containerName="maas-api" containerID="cri-o://a61dbb6846674ff966b442630e12c58d90fa5383ca2837caf90c6506f11d6622" gracePeriod=30 Aug 04 00:37:49.295923 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.295892 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7"] Aug 04 00:37:49.296151 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.296122 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" podUID="4e9fc144-5c2d-4d8e-91f2-62fc5241927b" containerName="payload-processing" containerID="cri-o://1a8685dd163792af484aa7f814f5e1179384b4d8f44f7c483853f29f6d67a27e" gracePeriod=30 Aug 04 00:37:49.314953 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.314254 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5"] Aug 04 00:37:49.316684 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.315901 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" podUID="a655be2d-8d01-4a67-bba3-3620d3e3a321" containerName="payload-pre-processing" containerID="cri-o://d3704dc526675a5937eab204a6ce30d7f84f6bd43f8cd07df2e63e32d68e2958" gracePeriod=30 Aug 04 00:37:49.528958 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.528930 2571 generic.go:356] "Generic (PLEG): container finished" podID="a655be2d-8d01-4a67-bba3-3620d3e3a321" containerID="d3704dc526675a5937eab204a6ce30d7f84f6bd43f8cd07df2e63e32d68e2958" exitCode=0 Aug 04 00:37:49.529120 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.529010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" event={"ID":"a655be2d-8d01-4a67-bba3-3620d3e3a321","Type":"ContainerDied","Data":"d3704dc526675a5937eab204a6ce30d7f84f6bd43f8cd07df2e63e32d68e2958"} Aug 04 00:37:49.530765 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.530744 2571 generic.go:356] "Generic (PLEG): container finished" podID="43d03ed0-2f57-4b7a-a60a-0cc98309107d" containerID="a61dbb6846674ff966b442630e12c58d90fa5383ca2837caf90c6506f11d6622" exitCode=0 Aug 04 00:37:49.530877 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.530816 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" event={"ID":"43d03ed0-2f57-4b7a-a60a-0cc98309107d","Type":"ContainerDied","Data":"a61dbb6846674ff966b442630e12c58d90fa5383ca2837caf90c6506f11d6622"} Aug 04 00:37:49.532476 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.532450 2571 generic.go:356] "Generic (PLEG): container finished" podID="4e9fc144-5c2d-4d8e-91f2-62fc5241927b" containerID="1a8685dd163792af484aa7f814f5e1179384b4d8f44f7c483853f29f6d67a27e" exitCode=0 Aug 04 00:37:49.532587 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.532485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" event={"ID":"4e9fc144-5c2d-4d8e-91f2-62fc5241927b","Type":"ContainerDied","Data":"1a8685dd163792af484aa7f814f5e1179384b4d8f44f7c483853f29f6d67a27e"} Aug 04 00:37:49.594237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.594197 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:49.614747 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.614665 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:49.635622 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.635599 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls\") pod \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " Aug 04 00:37:49.635762 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.635677 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/43d03ed0-2f57-4b7a-a60a-0cc98309107d-kube-api-access-k529k\") pod \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " Aug 04 00:37:49.635762 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.635696 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d03ed0-2f57-4b7a-a60a-0cc98309107d-openshift-service-ca\") pod \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\" (UID: \"43d03ed0-2f57-4b7a-a60a-0cc98309107d\") " Aug 04 00:37:49.636066 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.636035 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d03ed0-2f57-4b7a-a60a-0cc98309107d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43d03ed0-2f57-4b7a-a60a-0cc98309107d" (UID: "43d03ed0-2f57-4b7a-a60a-0cc98309107d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:49.636520 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.636498 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:49.637653 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.637629 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43d03ed0-2f57-4b7a-a60a-0cc98309107d" (UID: "43d03ed0-2f57-4b7a-a60a-0cc98309107d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:37:49.637743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.637707 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d03ed0-2f57-4b7a-a60a-0cc98309107d-kube-api-access-k529k" (OuterVolumeSpecName: "kube-api-access-k529k") pod "43d03ed0-2f57-4b7a-a60a-0cc98309107d" (UID: "43d03ed0-2f57-4b7a-a60a-0cc98309107d"). InnerVolumeSpecName "kube-api-access-k529k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:49.736575 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736541 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcklv\" (UniqueName: \"kubernetes.io/projected/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-kube-api-access-lcklv\") pod \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " Aug 04 00:37:49.736781 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736587 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdg5w\" (UniqueName: \"kubernetes.io/projected/a655be2d-8d01-4a67-bba3-3620d3e3a321-kube-api-access-fdg5w\") pod \"a655be2d-8d01-4a67-bba3-3620d3e3a321\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " Aug 04 00:37:49.736781 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736655 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a655be2d-8d01-4a67-bba3-3620d3e3a321-plugins-config-volume\") pod \"a655be2d-8d01-4a67-bba3-3620d3e3a321\" (UID: \"a655be2d-8d01-4a67-bba3-3620d3e3a321\") " Aug 04 00:37:49.736781 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736675 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-plugins-config-volume\") pod \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\" (UID: \"4e9fc144-5c2d-4d8e-91f2-62fc5241927b\") " Aug 04 00:37:49.736961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736859 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d03ed0-2f57-4b7a-a60a-0cc98309107d-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:49.736961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736877 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/43d03ed0-2f57-4b7a-a60a-0cc98309107d-kube-api-access-k529k\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:49.736961 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.736893 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d03ed0-2f57-4b7a-a60a-0cc98309107d-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:49.737092 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.737042 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a655be2d-8d01-4a67-bba3-3620d3e3a321-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a655be2d-8d01-4a67-bba3-3620d3e3a321" (UID: "a655be2d-8d01-4a67-bba3-3620d3e3a321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:49.737163 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.737084 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e9fc144-5c2d-4d8e-91f2-62fc5241927b" (UID: "4e9fc144-5c2d-4d8e-91f2-62fc5241927b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:37:49.738821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.738766 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a655be2d-8d01-4a67-bba3-3620d3e3a321-kube-api-access-fdg5w" (OuterVolumeSpecName: "kube-api-access-fdg5w") pod "a655be2d-8d01-4a67-bba3-3620d3e3a321" (UID: "a655be2d-8d01-4a67-bba3-3620d3e3a321"). InnerVolumeSpecName "kube-api-access-fdg5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:49.738821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.738778 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-kube-api-access-lcklv" (OuterVolumeSpecName: "kube-api-access-lcklv") pod "4e9fc144-5c2d-4d8e-91f2-62fc5241927b" (UID: "4e9fc144-5c2d-4d8e-91f2-62fc5241927b"). InnerVolumeSpecName "kube-api-access-lcklv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:37:49.837469 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.837439 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a655be2d-8d01-4a67-bba3-3620d3e3a321-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:49.837469 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.837465 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:49.837469 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.837474 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcklv\" (UniqueName: \"kubernetes.io/projected/4e9fc144-5c2d-4d8e-91f2-62fc5241927b-kube-api-access-lcklv\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:49.837716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:49.837483 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdg5w\" (UniqueName: \"kubernetes.io/projected/a655be2d-8d01-4a67-bba3-3620d3e3a321-kube-api-access-fdg5w\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:37:50.538338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.538311 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" Aug 04 00:37:50.538820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.538315 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6" event={"ID":"43d03ed0-2f57-4b7a-a60a-0cc98309107d","Type":"ContainerDied","Data":"560b4a124eb48c8b18c568bdcc12f67459c0d803784957dbe5352c6f3c503c94"} Aug 04 00:37:50.538820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.538450 2571 scope.go:117] "RemoveContainer" containerID="a61dbb6846674ff966b442630e12c58d90fa5383ca2837caf90c6506f11d6622" Aug 04 00:37:50.539842 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.539826 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" Aug 04 00:37:50.539923 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.539836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7" event={"ID":"4e9fc144-5c2d-4d8e-91f2-62fc5241927b","Type":"ContainerDied","Data":"3ce9920c1d9a29e02970090fb5e5f034380c5c8dd63fb73019ee2d2aa8cc28f1"} Aug 04 00:37:50.541447 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.541422 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" event={"ID":"a655be2d-8d01-4a67-bba3-3620d3e3a321","Type":"ContainerDied","Data":"89d3331f1a00553d0ba4bbaa7842b2bdc7a031e0cb45228ebff92689e683e844"} Aug 04 00:37:50.541560 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.541458 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5" Aug 04 00:37:50.548883 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.548868 2571 scope.go:117] "RemoveContainer" containerID="1a8685dd163792af484aa7f814f5e1179384b4d8f44f7c483853f29f6d67a27e" Aug 04 00:37:50.557415 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.556939 2571 scope.go:117] "RemoveContainer" containerID="d3704dc526675a5937eab204a6ce30d7f84f6bd43f8cd07df2e63e32d68e2958" Aug 04 00:37:50.559805 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.559780 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5"] Aug 04 00:37:50.570879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.570858 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fe5216a4-7498c994b8-twjz5"] Aug 04 00:37:50.585023 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.584999 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6"] Aug 04 00:37:50.589049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.589021 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe5216a4-59484b85fc-h42n6"] Aug 04 00:37:50.598978 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.598953 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7"] Aug 04 00:37:50.604539 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:50.604517 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fe5216a4-68b484f9d-hqtt7"] Aug 04 00:37:52.377001 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:52.376965 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d03ed0-2f57-4b7a-a60a-0cc98309107d" path="/var/lib/kubelet/pods/43d03ed0-2f57-4b7a-a60a-0cc98309107d/volumes" Aug 04 00:37:52.377408 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:52.377362 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9fc144-5c2d-4d8e-91f2-62fc5241927b" path="/var/lib/kubelet/pods/4e9fc144-5c2d-4d8e-91f2-62fc5241927b/volumes" Aug 04 00:37:52.377703 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:37:52.377688 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a655be2d-8d01-4a67-bba3-3620d3e3a321" path="/var/lib/kubelet/pods/a655be2d-8d01-4a67-bba3-3620d3e3a321/volumes" Aug 04 00:39:49.866049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866014 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk"] Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866422 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9fc144-5c2d-4d8e-91f2-62fc5241927b" containerName="payload-processing" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866433 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9fc144-5c2d-4d8e-91f2-62fc5241927b" containerName="payload-processing" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866443 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a655be2d-8d01-4a67-bba3-3620d3e3a321" containerName="payload-pre-processing" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866448 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a655be2d-8d01-4a67-bba3-3620d3e3a321" containerName="payload-pre-processing" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866460 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d03ed0-2f57-4b7a-a60a-0cc98309107d" containerName="maas-api" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866466 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d03ed0-2f57-4b7a-a60a-0cc98309107d" containerName="maas-api" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866516 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9fc144-5c2d-4d8e-91f2-62fc5241927b" containerName="payload-processing" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866526 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d03ed0-2f57-4b7a-a60a-0cc98309107d" containerName="maas-api" Aug 04 00:39:49.866579 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.866532 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a655be2d-8d01-4a67-bba3-3620d3e3a321" containerName="payload-pre-processing" Aug 04 00:39:49.869637 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.869620 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:49.871882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.871859 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4c743254-serving-cert\"" Aug 04 00:39:49.879476 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:49.879455 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk"] Aug 04 00:39:50.001692 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.001647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36051b1e-a628-4a2f-a001-1de0c64e4a62-openshift-service-ca\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.001897 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.001719 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.001897 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.001781 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9sp\" (UniqueName: \"kubernetes.io/projected/36051b1e-a628-4a2f-a001-1de0c64e4a62-kube-api-access-qw9sp\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.102881 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.102839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36051b1e-a628-4a2f-a001-1de0c64e4a62-openshift-service-ca\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.103086 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.102904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.103086 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.102949 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9sp\" (UniqueName: \"kubernetes.io/projected/36051b1e-a628-4a2f-a001-1de0c64e4a62-kube-api-access-qw9sp\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.103086 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:39:50.103069 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-serving-cert: secret "maas-api-e2e-mt-4c743254-serving-cert" not found Aug 04 00:39:50.103302 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:39:50.103180 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls podName:36051b1e-a628-4a2f-a001-1de0c64e4a62 nodeName:}" failed. No retries permitted until 2026-08-04 00:39:50.603157964 +0000 UTC m=+2228.774191261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls") pod "maas-api-e2e-mt-4c743254-7bd948dcc-56srk" (UID: "36051b1e-a628-4a2f-a001-1de0c64e4a62") : secret "maas-api-e2e-mt-4c743254-serving-cert" not found Aug 04 00:39:50.103515 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.103494 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36051b1e-a628-4a2f-a001-1de0c64e4a62-openshift-service-ca\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.111549 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.111526 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9sp\" (UniqueName: \"kubernetes.io/projected/36051b1e-a628-4a2f-a001-1de0c64e4a62-kube-api-access-qw9sp\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.521594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.521560 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f"] Aug 04 00:39:50.525039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.525023 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.527214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.527192 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4c743254\"" Aug 04 00:39:50.527326 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.527274 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4c743254-dockercfg-28fvs\"" Aug 04 00:39:50.535503 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.535471 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f"] Aug 04 00:39:50.607918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.607875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:50.608205 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:39:50.608062 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-serving-cert: secret "maas-api-e2e-mt-4c743254-serving-cert" not found Aug 04 00:39:50.608205 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:39:50.608159 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls podName:36051b1e-a628-4a2f-a001-1de0c64e4a62 nodeName:}" failed. No retries permitted until 2026-08-04 00:39:51.608138389 +0000 UTC m=+2229.779171689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls") pod "maas-api-e2e-mt-4c743254-7bd948dcc-56srk" (UID: "36051b1e-a628-4a2f-a001-1de0c64e4a62") : secret "maas-api-e2e-mt-4c743254-serving-cert" not found Aug 04 00:39:50.708675 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.708633 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhv96\" (UniqueName: \"kubernetes.io/projected/fb250768-300d-4c62-ae3f-510902f10b8a-kube-api-access-xhv96\") pod \"payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.708865 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.708687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb250768-300d-4c62-ae3f-510902f10b8a-plugins-config-volume\") pod \"payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.810095 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.810007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhv96\" (UniqueName: \"kubernetes.io/projected/fb250768-300d-4c62-ae3f-510902f10b8a-kube-api-access-xhv96\") pod \"payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.810095 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.810052 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb250768-300d-4c62-ae3f-510902f10b8a-plugins-config-volume\") pod \"payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.810765 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.810746 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb250768-300d-4c62-ae3f-510902f10b8a-plugins-config-volume\") pod \"payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.818303 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.818280 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhv96\" (UniqueName: \"kubernetes.io/projected/fb250768-300d-4c62-ae3f-510902f10b8a-kube-api-access-xhv96\") pod \"payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.836226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.836201 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:50.970763 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:50.970735 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f"] Aug 04 00:39:50.972738 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:39:50.972702 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb250768_300d_4c62_ae3f_510902f10b8a.slice/crio-a87a91ce54b8d74cc8ca371d0ffeb168b153d4489f0687446d9f0dee48975e9f WatchSource:0}: Error finding container a87a91ce54b8d74cc8ca371d0ffeb168b153d4489f0687446d9f0dee48975e9f: Status 404 returned error can't find the container with id a87a91ce54b8d74cc8ca371d0ffeb168b153d4489f0687446d9f0dee48975e9f Aug 04 00:39:51.180119 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.180016 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" event={"ID":"fb250768-300d-4c62-ae3f-510902f10b8a","Type":"ContainerStarted","Data":"60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8"} Aug 04 00:39:51.180293 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.180130 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" event={"ID":"fb250768-300d-4c62-ae3f-510902f10b8a","Type":"ContainerStarted","Data":"a87a91ce54b8d74cc8ca371d0ffeb168b153d4489f0687446d9f0dee48975e9f"} Aug 04 00:39:51.180293 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.180162 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:39:51.198470 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.198422 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" podStartSLOduration=1.198408285 podStartE2EDuration="1.198408285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:39:51.196036101 +0000 UTC m=+2229.367069409" watchObservedRunningTime="2026-08-04 00:39:51.198408285 +0000 UTC m=+2229.369441592" Aug 04 00:39:51.619723 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.619686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:51.622239 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.622210 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") pod \"maas-api-e2e-mt-4c743254-7bd948dcc-56srk\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:51.682135 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.682089 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:51.817214 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:51.817187 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk"] Aug 04 00:39:51.819311 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:39:51.819269 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36051b1e_a628_4a2f_a001_1de0c64e4a62.slice/crio-e9a9efc37ff0bdc6e30e0cdf5926d2318c6d1636afb793b2569d8bfdb6431337 WatchSource:0}: Error finding container e9a9efc37ff0bdc6e30e0cdf5926d2318c6d1636afb793b2569d8bfdb6431337: Status 404 returned error can't find the container with id e9a9efc37ff0bdc6e30e0cdf5926d2318c6d1636afb793b2569d8bfdb6431337 Aug 04 00:39:52.187182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:52.187142 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" event={"ID":"36051b1e-a628-4a2f-a001-1de0c64e4a62","Type":"ContainerStarted","Data":"e9a9efc37ff0bdc6e30e0cdf5926d2318c6d1636afb793b2569d8bfdb6431337"} Aug 04 00:39:53.194416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:53.194316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" event={"ID":"36051b1e-a628-4a2f-a001-1de0c64e4a62","Type":"ContainerStarted","Data":"c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac"} Aug 04 00:39:53.194416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:53.194389 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:39:53.210716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:53.210670 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" podStartSLOduration=3.24717087 podStartE2EDuration="4.210643481s" totalImagesPullingTime="963.472611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:39:53.209081691 +0000 UTC m=+2231.380114997" watchObservedRunningTime="2026-08-04 00:39:53.210643481 +0000 UTC m=+2231.381676788" Aug 04 00:39:59.205267 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:39:59.205234 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:40:02.189865 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:02.189831 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:40:31.618759 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:31.618721 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk"] Aug 04 00:40:31.619590 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:31.619548 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" podUID="36051b1e-a628-4a2f-a001-1de0c64e4a62" containerName="maas-api" containerID="cri-o://c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac" gracePeriod=30 Aug 04 00:40:31.675955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:31.675924 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f"] Aug 04 00:40:31.676173 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:31.676145 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" podUID="fb250768-300d-4c62-ae3f-510902f10b8a" containerName="payload-processing" containerID="cri-o://60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8" gracePeriod=30 Aug 04 00:40:31.936240 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:31.936215 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:40:31.958071 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:31.958047 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:40:32.078696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.078663 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb250768-300d-4c62-ae3f-510902f10b8a-plugins-config-volume\") pod \"fb250768-300d-4c62-ae3f-510902f10b8a\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " Aug 04 00:40:32.078890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.078719 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") pod \"36051b1e-a628-4a2f-a001-1de0c64e4a62\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " Aug 04 00:40:32.078890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.078751 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw9sp\" (UniqueName: \"kubernetes.io/projected/36051b1e-a628-4a2f-a001-1de0c64e4a62-kube-api-access-qw9sp\") pod \"36051b1e-a628-4a2f-a001-1de0c64e4a62\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " Aug 04 00:40:32.078890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.078775 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhv96\" (UniqueName: \"kubernetes.io/projected/fb250768-300d-4c62-ae3f-510902f10b8a-kube-api-access-xhv96\") pod \"fb250768-300d-4c62-ae3f-510902f10b8a\" (UID: \"fb250768-300d-4c62-ae3f-510902f10b8a\") " Aug 04 00:40:32.078890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.078832 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36051b1e-a628-4a2f-a001-1de0c64e4a62-openshift-service-ca\") pod \"36051b1e-a628-4a2f-a001-1de0c64e4a62\" (UID: \"36051b1e-a628-4a2f-a001-1de0c64e4a62\") " Aug 04 00:40:32.079271 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.079241 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb250768-300d-4c62-ae3f-510902f10b8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb250768-300d-4c62-ae3f-510902f10b8a" (UID: "fb250768-300d-4c62-ae3f-510902f10b8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:40:32.079352 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.079265 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36051b1e-a628-4a2f-a001-1de0c64e4a62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36051b1e-a628-4a2f-a001-1de0c64e4a62" (UID: "36051b1e-a628-4a2f-a001-1de0c64e4a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:40:32.080968 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.080937 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36051b1e-a628-4a2f-a001-1de0c64e4a62" (UID: "36051b1e-a628-4a2f-a001-1de0c64e4a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:40:32.080968 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.080957 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb250768-300d-4c62-ae3f-510902f10b8a-kube-api-access-xhv96" (OuterVolumeSpecName: "kube-api-access-xhv96") pod "fb250768-300d-4c62-ae3f-510902f10b8a" (UID: "fb250768-300d-4c62-ae3f-510902f10b8a"). InnerVolumeSpecName "kube-api-access-xhv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:40:32.081138 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.080984 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36051b1e-a628-4a2f-a001-1de0c64e4a62-kube-api-access-qw9sp" (OuterVolumeSpecName: "kube-api-access-qw9sp") pod "36051b1e-a628-4a2f-a001-1de0c64e4a62" (UID: "36051b1e-a628-4a2f-a001-1de0c64e4a62"). InnerVolumeSpecName "kube-api-access-qw9sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:40:32.179788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.179710 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36051b1e-a628-4a2f-a001-1de0c64e4a62-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.179788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.179736 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb250768-300d-4c62-ae3f-510902f10b8a-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.179788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.179746 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36051b1e-a628-4a2f-a001-1de0c64e4a62-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.179788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.179755 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw9sp\" (UniqueName: \"kubernetes.io/projected/36051b1e-a628-4a2f-a001-1de0c64e4a62-kube-api-access-qw9sp\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.179788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.179765 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhv96\" (UniqueName: \"kubernetes.io/projected/fb250768-300d-4c62-ae3f-510902f10b8a-kube-api-access-xhv96\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:40:32.407145 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.407097 2571 generic.go:356] "Generic (PLEG): container finished" podID="fb250768-300d-4c62-ae3f-510902f10b8a" containerID="60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8" exitCode=0 Aug 04 00:40:32.407298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.407148 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" Aug 04 00:40:32.407298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.407189 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" event={"ID":"fb250768-300d-4c62-ae3f-510902f10b8a","Type":"ContainerDied","Data":"60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8"} Aug 04 00:40:32.407298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.407242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f" event={"ID":"fb250768-300d-4c62-ae3f-510902f10b8a","Type":"ContainerDied","Data":"a87a91ce54b8d74cc8ca371d0ffeb168b153d4489f0687446d9f0dee48975e9f"} Aug 04 00:40:32.407298 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.407271 2571 scope.go:117] "RemoveContainer" containerID="60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8" Aug 04 00:40:32.408803 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.408779 2571 generic.go:356] "Generic (PLEG): container finished" podID="36051b1e-a628-4a2f-a001-1de0c64e4a62" containerID="c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac" exitCode=0 Aug 04 00:40:32.408911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.408834 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" event={"ID":"36051b1e-a628-4a2f-a001-1de0c64e4a62","Type":"ContainerDied","Data":"c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac"} Aug 04 00:40:32.408911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.408878 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" event={"ID":"36051b1e-a628-4a2f-a001-1de0c64e4a62","Type":"ContainerDied","Data":"e9a9efc37ff0bdc6e30e0cdf5926d2318c6d1636afb793b2569d8bfdb6431337"} Aug 04 00:40:32.408911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.408840 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk" Aug 04 00:40:32.416929 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.416908 2571 scope.go:117] "RemoveContainer" containerID="60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8" Aug 04 00:40:32.417246 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:40:32.417225 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8\": container with ID starting with 60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8 not found: ID does not exist" containerID="60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8" Aug 04 00:40:32.417348 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.417257 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8"} err="failed to get container status \"60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8\": rpc error: code = NotFound desc = could not find container \"60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8\": container with ID starting with 60c6fcbf7fc38baa9b963a33b5bcd841f9fe10fa9074f1bdb165e70966e2a9f8 not found: ID does not exist" Aug 04 00:40:32.417348 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.417283 2571 scope.go:117] "RemoveContainer" containerID="c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac" Aug 04 00:40:32.425940 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.425923 2571 scope.go:117] "RemoveContainer" containerID="c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac" Aug 04 00:40:32.426216 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:40:32.426194 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac\": container with ID starting with c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac not found: ID does not exist" containerID="c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac" Aug 04 00:40:32.426289 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.426220 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac"} err="failed to get container status \"c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac\": rpc error: code = NotFound desc = could not find container \"c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac\": container with ID starting with c5b4aeeaea23f3f89a28ad14a15e5548f3e4c1d596dafd77a930987c397f8dac not found: ID does not exist" Aug 04 00:40:32.430315 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.430269 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f"] Aug 04 00:40:32.434170 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.434150 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4c743254-58fb86c8cd-2bx9f"] Aug 04 00:40:32.444428 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.444404 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk"] Aug 04 00:40:32.449458 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:32.449438 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c743254-7bd948dcc-56srk"] Aug 04 00:40:34.376350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:34.376312 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36051b1e-a628-4a2f-a001-1de0c64e4a62" path="/var/lib/kubelet/pods/36051b1e-a628-4a2f-a001-1de0c64e4a62/volumes" Aug 04 00:40:34.376739 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:40:34.376697 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb250768-300d-4c62-ae3f-510902f10b8a" path="/var/lib/kubelet/pods/fb250768-300d-4c62-ae3f-510902f10b8a/volumes" Aug 04 00:42:57.844520 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.844488 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp"] Aug 04 00:42:57.845018 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.844999 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb250768-300d-4c62-ae3f-510902f10b8a" containerName="payload-processing" Aug 04 00:42:57.845067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.845020 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb250768-300d-4c62-ae3f-510902f10b8a" containerName="payload-processing" Aug 04 00:42:57.845067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.845031 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36051b1e-a628-4a2f-a001-1de0c64e4a62" containerName="maas-api" Aug 04 00:42:57.845067 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.845038 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="36051b1e-a628-4a2f-a001-1de0c64e4a62" containerName="maas-api" Aug 04 00:42:57.845220 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.845141 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb250768-300d-4c62-ae3f-510902f10b8a" containerName="payload-processing" Aug 04 00:42:57.845220 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.845150 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="36051b1e-a628-4a2f-a001-1de0c64e4a62" containerName="maas-api" Aug 04 00:42:57.848087 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.848068 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:57.850234 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.850214 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-746fe6d0-serving-cert\"" Aug 04 00:42:57.858255 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.858230 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp"] Aug 04 00:42:57.918849 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.918817 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffe59be-d958-41ed-a0c3-42b444ad4d93-maas-api-tls\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:57.919008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.918868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8lg\" (UniqueName: \"kubernetes.io/projected/5ffe59be-d958-41ed-a0c3-42b444ad4d93-kube-api-access-6g8lg\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:57.919008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:57.918901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffe59be-d958-41ed-a0c3-42b444ad4d93-openshift-service-ca\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.019362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.019324 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8lg\" (UniqueName: \"kubernetes.io/projected/5ffe59be-d958-41ed-a0c3-42b444ad4d93-kube-api-access-6g8lg\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.019362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.019367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffe59be-d958-41ed-a0c3-42b444ad4d93-openshift-service-ca\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.019599 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.019434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffe59be-d958-41ed-a0c3-42b444ad4d93-maas-api-tls\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.020042 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.020018 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffe59be-d958-41ed-a0c3-42b444ad4d93-openshift-service-ca\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.022139 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.022094 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffe59be-d958-41ed-a0c3-42b444ad4d93-maas-api-tls\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.027508 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.027482 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8lg\" (UniqueName: \"kubernetes.io/projected/5ffe59be-d958-41ed-a0c3-42b444ad4d93-kube-api-access-6g8lg\") pod \"maas-api-e2e-mt-746fe6d0-55485bd65f-54htp\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.159575 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.159488 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:58.297314 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.297281 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp"] Aug 04 00:42:58.299525 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:42:58.299499 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffe59be_d958_41ed_a0c3_42b444ad4d93.slice/crio-ee73c1b501f26127b416e457119b798aa7bb9ffbdb743c7d6f76841aca16338a WatchSource:0}: Error finding container ee73c1b501f26127b416e457119b798aa7bb9ffbdb743c7d6f76841aca16338a: Status 404 returned error can't find the container with id ee73c1b501f26127b416e457119b798aa7bb9ffbdb743c7d6f76841aca16338a Aug 04 00:42:58.301269 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.301250 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:42:58.508422 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.508336 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj"] Aug 04 00:42:58.513055 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.513038 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.515338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.515318 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-746fe6d0-dockercfg-92msl\"" Aug 04 00:42:58.515503 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.515485 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-746fe6d0\"" Aug 04 00:42:58.521807 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.521783 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj"] Aug 04 00:42:58.522548 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.522523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517947c8-6a67-4987-99ce-b839f269f90b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.522726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.522560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmb6b\" (UniqueName: \"kubernetes.io/projected/517947c8-6a67-4987-99ce-b839f269f90b-kube-api-access-wmb6b\") pod \"payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.528832 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.528810 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz"] Aug 04 00:42:58.532083 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.532066 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.545417 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.545392 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz"] Aug 04 00:42:58.623788 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.623746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9958098f-621d-41c6-994d-6159400cddf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.623955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.623821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmb6b\" (UniqueName: \"kubernetes.io/projected/517947c8-6a67-4987-99ce-b839f269f90b-kube-api-access-wmb6b\") pod \"payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.623955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.623944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517947c8-6a67-4987-99ce-b839f269f90b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.624091 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.623967 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxfg5\" (UniqueName: \"kubernetes.io/projected/9958098f-621d-41c6-994d-6159400cddf3-kube-api-access-xxfg5\") pod \"payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.624650 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.624631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517947c8-6a67-4987-99ce-b839f269f90b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.632743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.632710 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmb6b\" (UniqueName: \"kubernetes.io/projected/517947c8-6a67-4987-99ce-b839f269f90b-kube-api-access-wmb6b\") pod \"payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.724313 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.724271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9958098f-621d-41c6-994d-6159400cddf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.724503 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.724391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxfg5\" (UniqueName: \"kubernetes.io/projected/9958098f-621d-41c6-994d-6159400cddf3-kube-api-access-xxfg5\") pod \"payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.724975 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.724955 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9958098f-621d-41c6-994d-6159400cddf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.734937 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.734912 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxfg5\" (UniqueName: \"kubernetes.io/projected/9958098f-621d-41c6-994d-6159400cddf3-kube-api-access-xxfg5\") pod \"payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.824813 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.824783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:58.844978 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.844947 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:58.978314 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:58.978278 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj"] Aug 04 00:42:58.981777 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:42:58.981744 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517947c8_6a67_4987_99ce_b839f269f90b.slice/crio-6b6b2d21e99f0904bce7d89670a95951895ac06c7ef4c1d374c6d6a93088a963 WatchSource:0}: Error finding container 6b6b2d21e99f0904bce7d89670a95951895ac06c7ef4c1d374c6d6a93088a963: Status 404 returned error can't find the container with id 6b6b2d21e99f0904bce7d89670a95951895ac06c7ef4c1d374c6d6a93088a963 Aug 04 00:42:59.005226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.004790 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz"] Aug 04 00:42:59.165383 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.165344 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" event={"ID":"517947c8-6a67-4987-99ce-b839f269f90b","Type":"ContainerStarted","Data":"c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26"} Aug 04 00:42:59.165568 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.165413 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" event={"ID":"517947c8-6a67-4987-99ce-b839f269f90b","Type":"ContainerStarted","Data":"6b6b2d21e99f0904bce7d89670a95951895ac06c7ef4c1d374c6d6a93088a963"} Aug 04 00:42:59.165568 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.165551 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:42:59.167245 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.167214 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" event={"ID":"5ffe59be-d958-41ed-a0c3-42b444ad4d93","Type":"ContainerStarted","Data":"e86a18a2799d68b10aedb1becd9e38e0a7221878cfcc8afeb3ce357d7ed3ef64"} Aug 04 00:42:59.167394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.167268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" event={"ID":"5ffe59be-d958-41ed-a0c3-42b444ad4d93","Type":"ContainerStarted","Data":"ee73c1b501f26127b416e457119b798aa7bb9ffbdb743c7d6f76841aca16338a"} Aug 04 00:42:59.167394 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.167297 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:42:59.169155 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.169128 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" event={"ID":"9958098f-621d-41c6-994d-6159400cddf3","Type":"ContainerStarted","Data":"269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51"} Aug 04 00:42:59.169292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.169175 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" event={"ID":"9958098f-621d-41c6-994d-6159400cddf3","Type":"ContainerStarted","Data":"3d2486eb4b6d65b1745a809ccde25cd49cf4fa3ee055d860590070cb43e48f18"} Aug 04 00:42:59.169292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.169202 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:42:59.181607 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.181396 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" podStartSLOduration=1.181379122 podStartE2EDuration="1.181379122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:42:59.180706852 +0000 UTC m=+2417.351740156" watchObservedRunningTime="2026-08-04 00:42:59.181379122 +0000 UTC m=+2417.352412411" Aug 04 00:42:59.200209 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.200162 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" podStartSLOduration=1.647282167 podStartE2EDuration="2.20014523s" totalImagesPullingTime="552.863063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:42:59.198036397 +0000 UTC m=+2417.369069705" watchObservedRunningTime="2026-08-04 00:42:59.20014523 +0000 UTC m=+2417.371178584" Aug 04 00:42:59.214451 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:42:59.214405 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" podStartSLOduration=1.214391159 podStartE2EDuration="1.214391159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:42:59.212673446 +0000 UTC m=+2417.383706754" watchObservedRunningTime="2026-08-04 00:42:59.214391159 +0000 UTC m=+2417.385424464" Aug 04 00:43:05.182597 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:05.182516 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:43:10.178276 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:10.178246 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:43:10.178731 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:10.178323 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:43:55.839972 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:55.839935 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4"] Aug 04 00:43:55.843712 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:55.843689 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:43:55.846590 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:55.846212 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:43:55.849807 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:55.849785 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4"] Aug 04 00:43:55.927601 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:55.927566 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d5d09a-812f-4bd3-a2c2-606d57b07bfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4\" (UID: \"97d5d09a-812f-4bd3-a2c2-606d57b07bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:43:56.028575 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:56.028545 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d5d09a-812f-4bd3-a2c2-606d57b07bfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4\" (UID: \"97d5d09a-812f-4bd3-a2c2-606d57b07bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:43:56.029225 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:56.029205 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d5d09a-812f-4bd3-a2c2-606d57b07bfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4\" (UID: \"97d5d09a-812f-4bd3-a2c2-606d57b07bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:43:56.155987 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:56.155914 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:43:56.294824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:56.294795 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4"] Aug 04 00:43:56.296969 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:43:56.296944 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d5d09a_812f_4bd3_a2c2_606d57b07bfc.slice/crio-0360025de80741862097bf4523bf786bff0c22a2a51175c6d0bec5403ffa8234 WatchSource:0}: Error finding container 0360025de80741862097bf4523bf786bff0c22a2a51175c6d0bec5403ffa8234: Status 404 returned error can't find the container with id 0360025de80741862097bf4523bf786bff0c22a2a51175c6d0bec5403ffa8234 Aug 04 00:43:56.491189 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:43:56.491087 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" event={"ID":"97d5d09a-812f-4bd3-a2c2-606d57b07bfc","Type":"ContainerStarted","Data":"0360025de80741862097bf4523bf786bff0c22a2a51175c6d0bec5403ffa8234"} Aug 04 00:44:00.515979 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:00.515937 2571 generic.go:356] "Generic (PLEG): container finished" podID="97d5d09a-812f-4bd3-a2c2-606d57b07bfc" containerID="1cf639771bc9c6b78879c00c7465a2c09a42f3d6c7bb88e37062527f8f405854" exitCode=0 Aug 04 00:44:00.516398 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:00.515980 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" event={"ID":"97d5d09a-812f-4bd3-a2c2-606d57b07bfc","Type":"ContainerDied","Data":"1cf639771bc9c6b78879c00c7465a2c09a42f3d6c7bb88e37062527f8f405854"} Aug 04 00:44:01.656968 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:01.656941 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:44:01.679387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:01.679358 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d5d09a-812f-4bd3-a2c2-606d57b07bfc-maas-api-service-ca\") pod \"97d5d09a-812f-4bd3-a2c2-606d57b07bfc\" (UID: \"97d5d09a-812f-4bd3-a2c2-606d57b07bfc\") " Aug 04 00:44:01.679744 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:01.679718 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d5d09a-812f-4bd3-a2c2-606d57b07bfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97d5d09a-812f-4bd3-a2c2-606d57b07bfc" (UID: "97d5d09a-812f-4bd3-a2c2-606d57b07bfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:44:01.780449 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:01.780360 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d5d09a-812f-4bd3-a2c2-606d57b07bfc-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:02.527561 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:02.527518 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" event={"ID":"97d5d09a-812f-4bd3-a2c2-606d57b07bfc","Type":"ContainerDied","Data":"0360025de80741862097bf4523bf786bff0c22a2a51175c6d0bec5403ffa8234"} Aug 04 00:44:02.527766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:02.527586 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0360025de80741862097bf4523bf786bff0c22a2a51175c6d0bec5403ffa8234" Aug 04 00:44:02.527766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:02.527532 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4" Aug 04 00:44:11.435270 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.435231 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp"] Aug 04 00:44:11.435718 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.435489 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" podUID="5ffe59be-d958-41ed-a0c3-42b444ad4d93" containerName="maas-api" containerID="cri-o://e86a18a2799d68b10aedb1becd9e38e0a7221878cfcc8afeb3ce357d7ed3ef64" gracePeriod=30 Aug 04 00:44:11.487918 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.487891 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz"] Aug 04 00:44:11.488161 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.488110 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" podUID="9958098f-621d-41c6-994d-6159400cddf3" containerName="payload-processing" containerID="cri-o://269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51" gracePeriod=30 Aug 04 00:44:11.497861 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.497832 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj"] Aug 04 00:44:11.498254 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.498222 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" podUID="517947c8-6a67-4987-99ce-b839f269f90b" containerName="payload-pre-processing" containerID="cri-o://c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26" gracePeriod=30 Aug 04 00:44:11.576727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.576699 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ffe59be-d958-41ed-a0c3-42b444ad4d93" containerID="e86a18a2799d68b10aedb1becd9e38e0a7221878cfcc8afeb3ce357d7ed3ef64" exitCode=0 Aug 04 00:44:11.576854 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.576779 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" event={"ID":"5ffe59be-d958-41ed-a0c3-42b444ad4d93","Type":"ContainerDied","Data":"e86a18a2799d68b10aedb1becd9e38e0a7221878cfcc8afeb3ce357d7ed3ef64"} Aug 04 00:44:11.639498 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.639409 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4"] Aug 04 00:44:11.643269 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.643235 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-746fe6d0-5jjp4"] Aug 04 00:44:11.786027 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.786001 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:44:11.789051 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.789024 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:44:11.807892 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.807869 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:44:11.872700 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872669 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxfg5\" (UniqueName: \"kubernetes.io/projected/9958098f-621d-41c6-994d-6159400cddf3-kube-api-access-xxfg5\") pod \"9958098f-621d-41c6-994d-6159400cddf3\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " Aug 04 00:44:11.872872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872725 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmb6b\" (UniqueName: \"kubernetes.io/projected/517947c8-6a67-4987-99ce-b839f269f90b-kube-api-access-wmb6b\") pod \"517947c8-6a67-4987-99ce-b839f269f90b\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " Aug 04 00:44:11.872872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872756 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517947c8-6a67-4987-99ce-b839f269f90b-plugins-config-volume\") pod \"517947c8-6a67-4987-99ce-b839f269f90b\" (UID: \"517947c8-6a67-4987-99ce-b839f269f90b\") " Aug 04 00:44:11.872872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872783 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffe59be-d958-41ed-a0c3-42b444ad4d93-maas-api-tls\") pod \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " Aug 04 00:44:11.872872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872813 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9958098f-621d-41c6-994d-6159400cddf3-plugins-config-volume\") pod \"9958098f-621d-41c6-994d-6159400cddf3\" (UID: \"9958098f-621d-41c6-994d-6159400cddf3\") " Aug 04 00:44:11.872872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872847 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffe59be-d958-41ed-a0c3-42b444ad4d93-openshift-service-ca\") pod \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " Aug 04 00:44:11.873149 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.872908 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g8lg\" (UniqueName: \"kubernetes.io/projected/5ffe59be-d958-41ed-a0c3-42b444ad4d93-kube-api-access-6g8lg\") pod \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\" (UID: \"5ffe59be-d958-41ed-a0c3-42b444ad4d93\") " Aug 04 00:44:11.873238 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.873205 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9958098f-621d-41c6-994d-6159400cddf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9958098f-621d-41c6-994d-6159400cddf3" (UID: "9958098f-621d-41c6-994d-6159400cddf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:44:11.873290 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.873233 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517947c8-6a67-4987-99ce-b839f269f90b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "517947c8-6a67-4987-99ce-b839f269f90b" (UID: "517947c8-6a67-4987-99ce-b839f269f90b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:44:11.873337 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.873271 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffe59be-d958-41ed-a0c3-42b444ad4d93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ffe59be-d958-41ed-a0c3-42b444ad4d93" (UID: "5ffe59be-d958-41ed-a0c3-42b444ad4d93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:44:11.875038 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.875009 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffe59be-d958-41ed-a0c3-42b444ad4d93-kube-api-access-6g8lg" (OuterVolumeSpecName: "kube-api-access-6g8lg") pod "5ffe59be-d958-41ed-a0c3-42b444ad4d93" (UID: "5ffe59be-d958-41ed-a0c3-42b444ad4d93"). InnerVolumeSpecName "kube-api-access-6g8lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:44:11.875038 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.875017 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517947c8-6a67-4987-99ce-b839f269f90b-kube-api-access-wmb6b" (OuterVolumeSpecName: "kube-api-access-wmb6b") pod "517947c8-6a67-4987-99ce-b839f269f90b" (UID: "517947c8-6a67-4987-99ce-b839f269f90b"). InnerVolumeSpecName "kube-api-access-wmb6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:44:11.875311 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.875291 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ffe59be-d958-41ed-a0c3-42b444ad4d93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ffe59be-d958-41ed-a0c3-42b444ad4d93" (UID: "5ffe59be-d958-41ed-a0c3-42b444ad4d93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:44:11.875369 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.875292 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9958098f-621d-41c6-994d-6159400cddf3-kube-api-access-xxfg5" (OuterVolumeSpecName: "kube-api-access-xxfg5") pod "9958098f-621d-41c6-994d-6159400cddf3" (UID: "9958098f-621d-41c6-994d-6159400cddf3"). InnerVolumeSpecName "kube-api-access-xxfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973839 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8lg\" (UniqueName: \"kubernetes.io/projected/5ffe59be-d958-41ed-a0c3-42b444ad4d93-kube-api-access-6g8lg\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973864 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxfg5\" (UniqueName: \"kubernetes.io/projected/9958098f-621d-41c6-994d-6159400cddf3-kube-api-access-xxfg5\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973874 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmb6b\" (UniqueName: \"kubernetes.io/projected/517947c8-6a67-4987-99ce-b839f269f90b-kube-api-access-wmb6b\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973882 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517947c8-6a67-4987-99ce-b839f269f90b-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973891 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffe59be-d958-41ed-a0c3-42b444ad4d93-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973900 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9958098f-621d-41c6-994d-6159400cddf3-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:11.973911 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:11.973908 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffe59be-d958-41ed-a0c3-42b444ad4d93-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:44:12.376591 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.376552 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d5d09a-812f-4bd3-a2c2-606d57b07bfc" path="/var/lib/kubelet/pods/97d5d09a-812f-4bd3-a2c2-606d57b07bfc/volumes" Aug 04 00:44:12.583973 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.583937 2571 generic.go:356] "Generic (PLEG): container finished" podID="517947c8-6a67-4987-99ce-b839f269f90b" containerID="c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26" exitCode=0 Aug 04 00:44:12.584416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.583983 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" Aug 04 00:44:12.584416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.584011 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" event={"ID":"517947c8-6a67-4987-99ce-b839f269f90b","Type":"ContainerDied","Data":"c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26"} Aug 04 00:44:12.584416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.584064 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj" event={"ID":"517947c8-6a67-4987-99ce-b839f269f90b","Type":"ContainerDied","Data":"6b6b2d21e99f0904bce7d89670a95951895ac06c7ef4c1d374c6d6a93088a963"} Aug 04 00:44:12.584416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.584090 2571 scope.go:117] "RemoveContainer" containerID="c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26" Aug 04 00:44:12.585469 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.585450 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" Aug 04 00:44:12.585546 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.585506 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp" event={"ID":"5ffe59be-d958-41ed-a0c3-42b444ad4d93","Type":"ContainerDied","Data":"ee73c1b501f26127b416e457119b798aa7bb9ffbdb743c7d6f76841aca16338a"} Aug 04 00:44:12.587169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.587143 2571 generic.go:356] "Generic (PLEG): container finished" podID="9958098f-621d-41c6-994d-6159400cddf3" containerID="269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51" exitCode=0 Aug 04 00:44:12.587296 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.587195 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" Aug 04 00:44:12.587296 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.587229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" event={"ID":"9958098f-621d-41c6-994d-6159400cddf3","Type":"ContainerDied","Data":"269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51"} Aug 04 00:44:12.587296 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.587281 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz" event={"ID":"9958098f-621d-41c6-994d-6159400cddf3","Type":"ContainerDied","Data":"3d2486eb4b6d65b1745a809ccde25cd49cf4fa3ee055d860590070cb43e48f18"} Aug 04 00:44:12.594468 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.594438 2571 scope.go:117] "RemoveContainer" containerID="c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26" Aug 04 00:44:12.594706 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:44:12.594687 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26\": container with ID starting with c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26 not found: ID does not exist" containerID="c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26" Aug 04 00:44:12.594791 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.594719 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26"} err="failed to get container status \"c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26\": rpc error: code = NotFound desc = could not find container \"c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26\": container with ID starting with c6f2956e3bb4d5ff19126e849b331d7a4e7a4a674d6308fe9c850ae9c001ec26 not found: ID does not exist" Aug 04 00:44:12.594791 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.594741 2571 scope.go:117] "RemoveContainer" containerID="e86a18a2799d68b10aedb1becd9e38e0a7221878cfcc8afeb3ce357d7ed3ef64" Aug 04 00:44:12.602871 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.602840 2571 scope.go:117] "RemoveContainer" containerID="269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51" Aug 04 00:44:12.604459 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.604438 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp"] Aug 04 00:44:12.611231 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.611210 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-746fe6d0-55485bd65f-54htp"] Aug 04 00:44:12.612215 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.612200 2571 scope.go:117] "RemoveContainer" containerID="269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51" Aug 04 00:44:12.612454 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:44:12.612435 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51\": container with ID starting with 269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51 not found: ID does not exist" containerID="269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51" Aug 04 00:44:12.612523 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.612463 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51"} err="failed to get container status \"269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51\": rpc error: code = NotFound desc = could not find container \"269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51\": container with ID starting with 269337e5ad66845fb5043522ec7147f7999bc55c6766e5484d6be709d645cc51 not found: ID does not exist" Aug 04 00:44:12.618329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.618310 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj"] Aug 04 00:44:12.620358 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.620339 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-746fe6d0-7f6c5f4b96-cmbzj"] Aug 04 00:44:12.629237 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.629183 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz"] Aug 04 00:44:12.632890 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:12.632870 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-746fe6d0-697d6844cd-cvbdz"] Aug 04 00:44:14.377648 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:14.377610 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517947c8-6a67-4987-99ce-b839f269f90b" path="/var/lib/kubelet/pods/517947c8-6a67-4987-99ce-b839f269f90b/volumes" Aug 04 00:44:14.378064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:14.378011 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ffe59be-d958-41ed-a0c3-42b444ad4d93" path="/var/lib/kubelet/pods/5ffe59be-d958-41ed-a0c3-42b444ad4d93/volumes" Aug 04 00:44:14.378379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:44:14.378364 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9958098f-621d-41c6-994d-6159400cddf3" path="/var/lib/kubelet/pods/9958098f-621d-41c6-994d-6159400cddf3/volumes" Aug 04 00:45:00.141617 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.141578 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh"] Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.141960 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d5d09a-812f-4bd3-a2c2-606d57b07bfc" containerName="revoke-keys" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.141972 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d5d09a-812f-4bd3-a2c2-606d57b07bfc" containerName="revoke-keys" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.141987 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9958098f-621d-41c6-994d-6159400cddf3" containerName="payload-processing" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.141994 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9958098f-621d-41c6-994d-6159400cddf3" containerName="payload-processing" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142009 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ffe59be-d958-41ed-a0c3-42b444ad4d93" containerName="maas-api" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142017 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ffe59be-d958-41ed-a0c3-42b444ad4d93" containerName="maas-api" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142025 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517947c8-6a67-4987-99ce-b839f269f90b" containerName="payload-pre-processing" Aug 04 00:45:00.142053 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142031 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="517947c8-6a67-4987-99ce-b839f269f90b" containerName="payload-pre-processing" Aug 04 00:45:00.142350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142116 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ffe59be-d958-41ed-a0c3-42b444ad4d93" containerName="maas-api" Aug 04 00:45:00.142350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142129 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d5d09a-812f-4bd3-a2c2-606d57b07bfc" containerName="revoke-keys" Aug 04 00:45:00.142350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142135 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9958098f-621d-41c6-994d-6159400cddf3" containerName="payload-processing" Aug 04 00:45:00.142350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.142142 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="517947c8-6a67-4987-99ce-b839f269f90b" containerName="payload-pre-processing" Aug 04 00:45:00.145304 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.145286 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" Aug 04 00:45:00.147494 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.147468 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:45:00.151934 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.151904 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh"] Aug 04 00:45:00.157184 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.157165 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" Aug 04 00:45:00.293491 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.293461 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh"] Aug 04 00:45:00.295446 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:45:00.295410 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b444cfe_8337_44fd_a731_7f8eadc8fb38.slice/crio-7061df9b55718abd6d588475caab861363e933214be5d5e5647f8e08cbce9503 WatchSource:0}: Error finding container 7061df9b55718abd6d588475caab861363e933214be5d5e5647f8e08cbce9503: Status 404 returned error can't find the container with id 7061df9b55718abd6d588475caab861363e933214be5d5e5647f8e08cbce9503 Aug 04 00:45:00.840900 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.840861 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" event={"ID":"0b444cfe-8337-44fd-a731-7f8eadc8fb38","Type":"ContainerStarted","Data":"bb06434405fff3bed6218949684641732f8e9d3f6c4a8faae533a7e07ecb8b27"} Aug 04 00:45:00.841049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.840935 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" event={"ID":"0b444cfe-8337-44fd-a731-7f8eadc8fb38","Type":"ContainerStarted","Data":"7061df9b55718abd6d588475caab861363e933214be5d5e5647f8e08cbce9503"} Aug 04 00:45:00.857222 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:00.857172 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" podStartSLOduration=0.857155669 podStartE2EDuration="857.155669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:45:00.855775074 +0000 UTC m=+2539.026808402" watchObservedRunningTime="2026-08-04 00:45:00.857155669 +0000 UTC m=+2539.028188975" Aug 04 00:45:01.847797 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:01.847762 2571 generic.go:356] "Generic (PLEG): container finished" podID="0b444cfe-8337-44fd-a731-7f8eadc8fb38" containerID="bb06434405fff3bed6218949684641732f8e9d3f6c4a8faae533a7e07ecb8b27" exitCode=0 Aug 04 00:45:01.848206 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:01.847852 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" event={"ID":"0b444cfe-8337-44fd-a731-7f8eadc8fb38","Type":"ContainerDied","Data":"bb06434405fff3bed6218949684641732f8e9d3f6c4a8faae533a7e07ecb8b27"} Aug 04 00:45:03.008282 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:03.008256 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" Aug 04 00:45:03.860188 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:03.860155 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" Aug 04 00:45:03.860358 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:03.860158 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763405-njwfh" event={"ID":"0b444cfe-8337-44fd-a731-7f8eadc8fb38","Type":"ContainerDied","Data":"7061df9b55718abd6d588475caab861363e933214be5d5e5647f8e08cbce9503"} Aug 04 00:45:03.860358 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:45:03.860278 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7061df9b55718abd6d588475caab861363e933214be5d5e5647f8e08cbce9503" Aug 04 00:46:11.904843 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.904760 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z"] Aug 04 00:46:11.905452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.905307 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b444cfe-8337-44fd-a731-7f8eadc8fb38" containerName="cleanup" Aug 04 00:46:11.905452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.905327 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b444cfe-8337-44fd-a731-7f8eadc8fb38" containerName="cleanup" Aug 04 00:46:11.905452 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.905431 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b444cfe-8337-44fd-a731-7f8eadc8fb38" containerName="cleanup" Aug 04 00:46:11.908846 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.908827 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:11.911218 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.911194 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4f7c8d-serving-cert\"" Aug 04 00:46:11.920176 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:11.920152 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z"] Aug 04 00:46:12.010827 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.010792 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553e528a-ab77-4923-a78d-8bb91d495449-maas-api-tls\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.010989 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.010829 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbk8\" (UniqueName: \"kubernetes.io/projected/553e528a-ab77-4923-a78d-8bb91d495449-kube-api-access-5pbk8\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.010989 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.010931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553e528a-ab77-4923-a78d-8bb91d495449-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.112493 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.112456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553e528a-ab77-4923-a78d-8bb91d495449-maas-api-tls\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.112493 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.112499 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbk8\" (UniqueName: \"kubernetes.io/projected/553e528a-ab77-4923-a78d-8bb91d495449-kube-api-access-5pbk8\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.112707 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.112532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553e528a-ab77-4923-a78d-8bb91d495449-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.113280 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.113255 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553e528a-ab77-4923-a78d-8bb91d495449-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.114997 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.114972 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553e528a-ab77-4923-a78d-8bb91d495449-maas-api-tls\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.119749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.119730 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbk8\" (UniqueName: \"kubernetes.io/projected/553e528a-ab77-4923-a78d-8bb91d495449-kube-api-access-5pbk8\") pod \"maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.220564 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.220482 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:12.357007 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.356980 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z"] Aug 04 00:46:12.359866 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:46:12.359836 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553e528a_ab77_4923_a78d_8bb91d495449.slice/crio-557417cf3ae42eae219588332deabe51f4f55e00d833e1e4e8270947519370a4 WatchSource:0}: Error finding container 557417cf3ae42eae219588332deabe51f4f55e00d833e1e4e8270947519370a4: Status 404 returned error can't find the container with id 557417cf3ae42eae219588332deabe51f4f55e00d833e1e4e8270947519370a4 Aug 04 00:46:12.607991 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.607955 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg"] Aug 04 00:46:12.612726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.612704 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.617965 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.617946 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4f7c8d\"" Aug 04 00:46:12.618166 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.618145 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4f7c8d-dockercfg-lvc5w\"" Aug 04 00:46:12.624040 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.624011 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg"] Aug 04 00:46:12.639632 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.639586 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv"] Aug 04 00:46:12.643728 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.643709 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.651156 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.651050 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv"] Aug 04 00:46:12.717092 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.717053 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c493ee2-61c9-4caa-ab21-9c957e707a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.717292 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.717206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64nls\" (UniqueName: \"kubernetes.io/projected/1c493ee2-61c9-4caa-ab21-9c957e707a22-kube-api-access-64nls\") pod \"payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.817830 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.817788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtzf\" (UniqueName: \"kubernetes.io/projected/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-kube-api-access-hjtzf\") pod \"payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.818039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.817854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64nls\" (UniqueName: \"kubernetes.io/projected/1c493ee2-61c9-4caa-ab21-9c957e707a22-kube-api-access-64nls\") pod \"payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.818039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.817949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.818039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.817988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c493ee2-61c9-4caa-ab21-9c957e707a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.818680 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.818661 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c493ee2-61c9-4caa-ab21-9c957e707a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.825662 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.825638 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64nls\" (UniqueName: \"kubernetes.io/projected/1c493ee2-61c9-4caa-ab21-9c957e707a22-kube-api-access-64nls\") pod \"payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.919343 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.919315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.919671 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.919418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtzf\" (UniqueName: \"kubernetes.io/projected/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-kube-api-access-hjtzf\") pod \"payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.920080 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.920061 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.924799 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.924781 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:12.926566 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.926548 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtzf\" (UniqueName: \"kubernetes.io/projected/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-kube-api-access-hjtzf\") pod \"payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:12.955260 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:12.955228 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:13.075146 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.075115 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg"] Aug 04 00:46:13.078017 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:46:13.077982 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c493ee2_61c9_4caa_ab21_9c957e707a22.slice/crio-477276ca5b167f5d20c4b69188f366a5e2ace93eab268d40112efd80c18e420f WatchSource:0}: Error finding container 477276ca5b167f5d20c4b69188f366a5e2ace93eab268d40112efd80c18e420f: Status 404 returned error can't find the container with id 477276ca5b167f5d20c4b69188f366a5e2ace93eab268d40112efd80c18e420f Aug 04 00:46:13.105819 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.105793 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv"] Aug 04 00:46:13.112342 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:46:13.112317 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3158b3c9_6fc0_4bf6_8cc1_408a43766d4e.slice/crio-6f50a81122a0cde1dd9fcb3a34539d1d4f17acead1322ce2558eea1f4524444e WatchSource:0}: Error finding container 6f50a81122a0cde1dd9fcb3a34539d1d4f17acead1322ce2558eea1f4524444e: Status 404 returned error can't find the container with id 6f50a81122a0cde1dd9fcb3a34539d1d4f17acead1322ce2558eea1f4524444e Aug 04 00:46:13.219317 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.219279 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" event={"ID":"1c493ee2-61c9-4caa-ab21-9c957e707a22","Type":"ContainerStarted","Data":"835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf"} Aug 04 00:46:13.219550 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.219346 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" event={"ID":"1c493ee2-61c9-4caa-ab21-9c957e707a22","Type":"ContainerStarted","Data":"477276ca5b167f5d20c4b69188f366a5e2ace93eab268d40112efd80c18e420f"} Aug 04 00:46:13.219550 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.219496 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:13.220931 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.220908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" event={"ID":"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e","Type":"ContainerStarted","Data":"70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45"} Aug 04 00:46:13.221046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.220947 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:13.221046 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.220969 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" event={"ID":"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e","Type":"ContainerStarted","Data":"6f50a81122a0cde1dd9fcb3a34539d1d4f17acead1322ce2558eea1f4524444e"} Aug 04 00:46:13.222736 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.222700 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" event={"ID":"553e528a-ab77-4923-a78d-8bb91d495449","Type":"ContainerStarted","Data":"aa2366a03d4dfb60b48eeb1c240416307648a2de48687acdc9fd1bc107f30948"} Aug 04 00:46:13.222842 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.222753 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" event={"ID":"553e528a-ab77-4923-a78d-8bb91d495449","Type":"ContainerStarted","Data":"557417cf3ae42eae219588332deabe51f4f55e00d833e1e4e8270947519370a4"} Aug 04 00:46:13.222842 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.222788 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:13.234960 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.234918 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" podStartSLOduration=1.23490548 podStartE2EDuration="1.23490548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:13.232628392 +0000 UTC m=+2611.403661699" watchObservedRunningTime="2026-08-04 00:46:13.23490548 +0000 UTC m=+2611.405938765" Aug 04 00:46:13.248327 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.248285 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" podStartSLOduration=1.248271119 podStartE2EDuration="1.248271119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:13.246985326 +0000 UTC m=+2611.418018626" watchObservedRunningTime="2026-08-04 00:46:13.248271119 +0000 UTC m=+2611.419304426" Aug 04 00:46:13.265142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:13.265075 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" podStartSLOduration=1.724284955 podStartE2EDuration="2.265050164s" totalImagesPullingTime="540.765209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:13.264847188 +0000 UTC m=+2611.435880499" watchObservedRunningTime="2026-08-04 00:46:13.265050164 +0000 UTC m=+2611.436083471" Aug 04 00:46:19.241389 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:19.241351 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:46:24.237976 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:24.237930 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:46:24.238466 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:24.238021 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:46:39.944178 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:39.944138 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh"] Aug 04 00:46:39.952416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:39.952385 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:39.954481 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:39.954457 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh"] Aug 04 00:46:39.954851 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:39.954826 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-743d5d-serving-cert\"" Aug 04 00:46:40.035096 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.035061 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.035312 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.035121 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def84d82-5830-4fac-aad4-b54e9ecb2854-openshift-service-ca\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.035312 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.035226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kwg\" (UniqueName: \"kubernetes.io/projected/def84d82-5830-4fac-aad4-b54e9ecb2854-kube-api-access-r4kwg\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.136352 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.136311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def84d82-5830-4fac-aad4-b54e9ecb2854-openshift-service-ca\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.136522 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.136377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kwg\" (UniqueName: \"kubernetes.io/projected/def84d82-5830-4fac-aad4-b54e9ecb2854-kube-api-access-r4kwg\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.136522 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.136454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.136602 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:46:40.136554 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-serving-cert: secret "maas-api-e2e-inf-b-743d5d-serving-cert" not found Aug 04 00:46:40.136638 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:46:40.136619 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls podName:def84d82-5830-4fac-aad4-b54e9ecb2854 nodeName:}" failed. No retries permitted until 2026-08-04 00:46:40.636602266 +0000 UTC m=+2638.807635551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls") pod "maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" (UID: "def84d82-5830-4fac-aad4-b54e9ecb2854") : secret "maas-api-e2e-inf-b-743d5d-serving-cert" not found Aug 04 00:46:40.137009 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.136987 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def84d82-5830-4fac-aad4-b54e9ecb2854-openshift-service-ca\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.144421 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.144390 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kwg\" (UniqueName: \"kubernetes.io/projected/def84d82-5830-4fac-aad4-b54e9ecb2854-kube-api-access-r4kwg\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.640594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.640553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.642042 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.642003 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd"] Aug 04 00:46:40.643058 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.643036 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls\") pod \"maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.646178 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.646160 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.648487 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.648464 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-743d5d-dockercfg-zpqrs\"" Aug 04 00:46:40.648487 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.648484 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-743d5d\"" Aug 04 00:46:40.655929 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.655903 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd"] Aug 04 00:46:40.741982 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.741948 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vv9r\" (UniqueName: \"kubernetes.io/projected/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-kube-api-access-6vv9r\") pod \"payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.742166 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.741997 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.842756 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.842724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vv9r\" (UniqueName: \"kubernetes.io/projected/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-kube-api-access-6vv9r\") pod \"payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.842936 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.842786 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.843533 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.843513 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.850534 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.850507 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vv9r\" (UniqueName: \"kubernetes.io/projected/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-kube-api-access-6vv9r\") pod \"payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:40.866342 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.866316 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:40.962470 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:40.962435 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:41.003203 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.003168 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh"] Aug 04 00:46:41.004921 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:46:41.004873 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef84d82_5830_4fac_aad4_b54e9ecb2854.slice/crio-5e08b09ef7299625d22581412010406c0f45770c8ddb070885b2486f9dcc9461 WatchSource:0}: Error finding container 5e08b09ef7299625d22581412010406c0f45770c8ddb070885b2486f9dcc9461: Status 404 returned error can't find the container with id 5e08b09ef7299625d22581412010406c0f45770c8ddb070885b2486f9dcc9461 Aug 04 00:46:41.099649 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.099617 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd"] Aug 04 00:46:41.101441 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:46:41.101418 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda965b1f1_f5d0_4df0_b8ed_a45d0eb2992e.slice/crio-e11a2d4c4c9c40371f8e35bde79f9e0671d7e7e7eb4ce9b197de0211228faf57 WatchSource:0}: Error finding container e11a2d4c4c9c40371f8e35bde79f9e0671d7e7e7eb4ce9b197de0211228faf57: Status 404 returned error can't find the container with id e11a2d4c4c9c40371f8e35bde79f9e0671d7e7e7eb4ce9b197de0211228faf57 Aug 04 00:46:41.385380 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.385336 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" event={"ID":"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e","Type":"ContainerStarted","Data":"fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438"} Aug 04 00:46:41.385572 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.385409 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:46:41.385572 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.385429 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" event={"ID":"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e","Type":"ContainerStarted","Data":"e11a2d4c4c9c40371f8e35bde79f9e0671d7e7e7eb4ce9b197de0211228faf57"} Aug 04 00:46:41.386670 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.386645 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" event={"ID":"def84d82-5830-4fac-aad4-b54e9ecb2854","Type":"ContainerStarted","Data":"5e08b09ef7299625d22581412010406c0f45770c8ddb070885b2486f9dcc9461"} Aug 04 00:46:41.400117 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:41.400037 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" podStartSLOduration=1.40002051 podStartE2EDuration="1.40002051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:41.39928809 +0000 UTC m=+2639.570321398" watchObservedRunningTime="2026-08-04 00:46:41.40002051 +0000 UTC m=+2639.571053817" Aug 04 00:46:42.393039 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:42.392999 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" event={"ID":"def84d82-5830-4fac-aad4-b54e9ecb2854","Type":"ContainerStarted","Data":"6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5"} Aug 04 00:46:42.393480 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:42.393265 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:42.409304 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:42.409262 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" podStartSLOduration=2.762965538 podStartE2EDuration="3.409246088s" totalImagesPullingTime="646.28055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:46:42.408035892 +0000 UTC m=+2640.579069199" watchObservedRunningTime="2026-08-04 00:46:42.409246088 +0000 UTC m=+2640.580279423" Aug 04 00:46:48.404880 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:48.404847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:46:52.395714 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:46:52.395678 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:48:14.155893 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.155850 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg"] Aug 04 00:48:14.159569 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.159548 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:14.161821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.161801 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:48:14.167172 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.167143 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg"] Aug 04 00:48:14.269967 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.269930 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239ab6b-833c-49f8-867b-a9b0927f17fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg\" (UID: \"9239ab6b-833c-49f8-867b-a9b0927f17fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:14.371455 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.371419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239ab6b-833c-49f8-867b-a9b0927f17fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg\" (UID: \"9239ab6b-833c-49f8-867b-a9b0927f17fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:14.372031 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.372011 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239ab6b-833c-49f8-867b-a9b0927f17fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg\" (UID: \"9239ab6b-833c-49f8-867b-a9b0927f17fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:14.473141 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.473043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:14.612769 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.612743 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg"] Aug 04 00:48:14.614829 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:48:14.614801 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9239ab6b_833c_49f8_867b_a9b0927f17fa.slice/crio-d5ba14ae95cbcd81a70cf75f8376a7e735c5be773a157f05f2ec34d84010f15a WatchSource:0}: Error finding container d5ba14ae95cbcd81a70cf75f8376a7e735c5be773a157f05f2ec34d84010f15a: Status 404 returned error can't find the container with id d5ba14ae95cbcd81a70cf75f8376a7e735c5be773a157f05f2ec34d84010f15a Aug 04 00:48:14.900352 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:14.900317 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" event={"ID":"9239ab6b-833c-49f8-867b-a9b0927f17fa","Type":"ContainerStarted","Data":"d5ba14ae95cbcd81a70cf75f8376a7e735c5be773a157f05f2ec34d84010f15a"} Aug 04 00:48:15.908052 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:15.908014 2571 generic.go:356] "Generic (PLEG): container finished" podID="9239ab6b-833c-49f8-867b-a9b0927f17fa" containerID="7a3fd13b090dd7a9b07b9caa36b1a986e52197dc5457a09e40634e5e886fd707" exitCode=0 Aug 04 00:48:15.908456 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:15.908064 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" event={"ID":"9239ab6b-833c-49f8-867b-a9b0927f17fa","Type":"ContainerDied","Data":"7a3fd13b090dd7a9b07b9caa36b1a986e52197dc5457a09e40634e5e886fd707"} Aug 04 00:48:17.047221 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.047187 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:17.200287 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.200198 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239ab6b-833c-49f8-867b-a9b0927f17fa-maas-api-service-ca\") pod \"9239ab6b-833c-49f8-867b-a9b0927f17fa\" (UID: \"9239ab6b-833c-49f8-867b-a9b0927f17fa\") " Aug 04 00:48:17.200656 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.200631 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9239ab6b-833c-49f8-867b-a9b0927f17fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9239ab6b-833c-49f8-867b-a9b0927f17fa" (UID: "9239ab6b-833c-49f8-867b-a9b0927f17fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:17.301884 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.301844 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9239ab6b-833c-49f8-867b-a9b0927f17fa-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:17.920875 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.920844 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" Aug 04 00:48:17.921059 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.920843 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg" event={"ID":"9239ab6b-833c-49f8-867b-a9b0927f17fa","Type":"ContainerDied","Data":"d5ba14ae95cbcd81a70cf75f8376a7e735c5be773a157f05f2ec34d84010f15a"} Aug 04 00:48:17.921059 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:17.920967 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ba14ae95cbcd81a70cf75f8376a7e735c5be773a157f05f2ec34d84010f15a" Aug 04 00:48:18.750045 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.750006 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z"] Aug 04 00:48:18.750476 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.750295 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" podUID="553e528a-ab77-4923-a78d-8bb91d495449" containerName="maas-api" containerID="cri-o://aa2366a03d4dfb60b48eeb1c240416307648a2de48687acdc9fd1bc107f30948" gracePeriod=30 Aug 04 00:48:18.815562 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.815534 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv"] Aug 04 00:48:18.815789 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.815736 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" podUID="3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" containerName="payload-processing" containerID="cri-o://70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45" gracePeriod=30 Aug 04 00:48:18.827672 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.827453 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg"] Aug 04 00:48:18.827803 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.827682 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" podUID="1c493ee2-61c9-4caa-ab21-9c957e707a22" containerName="payload-pre-processing" containerID="cri-o://835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf" gracePeriod=30 Aug 04 00:48:18.929569 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.929536 2571 generic.go:356] "Generic (PLEG): container finished" podID="553e528a-ab77-4923-a78d-8bb91d495449" containerID="aa2366a03d4dfb60b48eeb1c240416307648a2de48687acdc9fd1bc107f30948" exitCode=0 Aug 04 00:48:18.929723 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.929605 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" event={"ID":"553e528a-ab77-4923-a78d-8bb91d495449","Type":"ContainerDied","Data":"aa2366a03d4dfb60b48eeb1c240416307648a2de48687acdc9fd1bc107f30948"} Aug 04 00:48:18.972976 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.972934 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg"] Aug 04 00:48:18.976985 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:18.976939 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f7c8d-gq4gg"] Aug 04 00:48:19.112261 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.112232 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:48:19.123252 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.123230 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:48:19.128422 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.128401 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:48:19.218855 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.218820 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c493ee2-61c9-4caa-ab21-9c957e707a22-plugins-config-volume\") pod \"1c493ee2-61c9-4caa-ab21-9c957e707a22\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " Aug 04 00:48:19.219049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.218884 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553e528a-ab77-4923-a78d-8bb91d495449-maas-api-tls\") pod \"553e528a-ab77-4923-a78d-8bb91d495449\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " Aug 04 00:48:19.219049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.218906 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjtzf\" (UniqueName: \"kubernetes.io/projected/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-kube-api-access-hjtzf\") pod \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " Aug 04 00:48:19.219049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.218947 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-plugins-config-volume\") pod \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\" (UID: \"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e\") " Aug 04 00:48:19.219049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.218973 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pbk8\" (UniqueName: \"kubernetes.io/projected/553e528a-ab77-4923-a78d-8bb91d495449-kube-api-access-5pbk8\") pod \"553e528a-ab77-4923-a78d-8bb91d495449\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " Aug 04 00:48:19.219049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.219031 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64nls\" (UniqueName: \"kubernetes.io/projected/1c493ee2-61c9-4caa-ab21-9c957e707a22-kube-api-access-64nls\") pod \"1c493ee2-61c9-4caa-ab21-9c957e707a22\" (UID: \"1c493ee2-61c9-4caa-ab21-9c957e707a22\") " Aug 04 00:48:19.219367 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.219054 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553e528a-ab77-4923-a78d-8bb91d495449-openshift-service-ca\") pod \"553e528a-ab77-4923-a78d-8bb91d495449\" (UID: \"553e528a-ab77-4923-a78d-8bb91d495449\") " Aug 04 00:48:19.219367 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.219313 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c493ee2-61c9-4caa-ab21-9c957e707a22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c493ee2-61c9-4caa-ab21-9c957e707a22" (UID: "1c493ee2-61c9-4caa-ab21-9c957e707a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:19.219367 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.219349 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" (UID: "3158b3c9-6fc0-4bf6-8cc1-408a43766d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:19.219689 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.219659 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553e528a-ab77-4923-a78d-8bb91d495449-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "553e528a-ab77-4923-a78d-8bb91d495449" (UID: "553e528a-ab77-4923-a78d-8bb91d495449"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:19.221182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.221153 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-kube-api-access-hjtzf" (OuterVolumeSpecName: "kube-api-access-hjtzf") pod "3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" (UID: "3158b3c9-6fc0-4bf6-8cc1-408a43766d4e"). InnerVolumeSpecName "kube-api-access-hjtzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:19.221297 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.221276 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553e528a-ab77-4923-a78d-8bb91d495449-kube-api-access-5pbk8" (OuterVolumeSpecName: "kube-api-access-5pbk8") pod "553e528a-ab77-4923-a78d-8bb91d495449" (UID: "553e528a-ab77-4923-a78d-8bb91d495449"). InnerVolumeSpecName "kube-api-access-5pbk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:19.221516 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.221493 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c493ee2-61c9-4caa-ab21-9c957e707a22-kube-api-access-64nls" (OuterVolumeSpecName: "kube-api-access-64nls") pod "1c493ee2-61c9-4caa-ab21-9c957e707a22" (UID: "1c493ee2-61c9-4caa-ab21-9c957e707a22"). InnerVolumeSpecName "kube-api-access-64nls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:19.221602 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.221497 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553e528a-ab77-4923-a78d-8bb91d495449-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "553e528a-ab77-4923-a78d-8bb91d495449" (UID: "553e528a-ab77-4923-a78d-8bb91d495449"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320063 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64nls\" (UniqueName: \"kubernetes.io/projected/1c493ee2-61c9-4caa-ab21-9c957e707a22-kube-api-access-64nls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320095 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553e528a-ab77-4923-a78d-8bb91d495449-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320134 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c493ee2-61c9-4caa-ab21-9c957e707a22-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320146 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553e528a-ab77-4923-a78d-8bb91d495449-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320158 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtzf\" (UniqueName: \"kubernetes.io/projected/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-kube-api-access-hjtzf\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320170 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.320181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.320182 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbk8\" (UniqueName: \"kubernetes.io/projected/553e528a-ab77-4923-a78d-8bb91d495449-kube-api-access-5pbk8\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:19.936320 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.936281 2571 generic.go:356] "Generic (PLEG): container finished" podID="1c493ee2-61c9-4caa-ab21-9c957e707a22" containerID="835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf" exitCode=0 Aug 04 00:48:19.936796 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.936327 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" Aug 04 00:48:19.936796 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.936405 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" event={"ID":"1c493ee2-61c9-4caa-ab21-9c957e707a22","Type":"ContainerDied","Data":"835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf"} Aug 04 00:48:19.936796 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.936454 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg" event={"ID":"1c493ee2-61c9-4caa-ab21-9c957e707a22","Type":"ContainerDied","Data":"477276ca5b167f5d20c4b69188f366a5e2ace93eab268d40112efd80c18e420f"} Aug 04 00:48:19.936796 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.936489 2571 scope.go:117] "RemoveContainer" containerID="835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf" Aug 04 00:48:19.937854 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.937827 2571 generic.go:356] "Generic (PLEG): container finished" podID="3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" containerID="70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45" exitCode=0 Aug 04 00:48:19.937955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.937864 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" Aug 04 00:48:19.937955 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.937897 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" event={"ID":"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e","Type":"ContainerDied","Data":"70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45"} Aug 04 00:48:19.938078 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.937962 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv" event={"ID":"3158b3c9-6fc0-4bf6-8cc1-408a43766d4e","Type":"ContainerDied","Data":"6f50a81122a0cde1dd9fcb3a34539d1d4f17acead1322ce2558eea1f4524444e"} Aug 04 00:48:19.939959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.939937 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" Aug 04 00:48:19.939959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.939946 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z" event={"ID":"553e528a-ab77-4923-a78d-8bb91d495449","Type":"ContainerDied","Data":"557417cf3ae42eae219588332deabe51f4f55e00d833e1e4e8270947519370a4"} Aug 04 00:48:19.947396 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.947008 2571 scope.go:117] "RemoveContainer" containerID="835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf" Aug 04 00:48:19.947396 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:48:19.947319 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf\": container with ID starting with 835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf not found: ID does not exist" containerID="835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf" Aug 04 00:48:19.947396 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.947343 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf"} err="failed to get container status \"835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf\": rpc error: code = NotFound desc = could not find container \"835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf\": container with ID starting with 835c45f261650bdd87b681be556eed83a4a908feda3c3bd96cb7679ea723b5cf not found: ID does not exist" Aug 04 00:48:19.947396 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.947366 2571 scope.go:117] "RemoveContainer" containerID="70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45" Aug 04 00:48:19.959276 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.959257 2571 scope.go:117] "RemoveContainer" containerID="70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45" Aug 04 00:48:19.959564 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:48:19.959546 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45\": container with ID starting with 70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45 not found: ID does not exist" containerID="70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45" Aug 04 00:48:19.959613 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.959571 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45"} err="failed to get container status \"70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45\": rpc error: code = NotFound desc = could not find container \"70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45\": container with ID starting with 70316534d29aeea64f6c793739f9b02c64629cf2bfb3a71528cdee463fd55d45 not found: ID does not exist" Aug 04 00:48:19.959613 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.959589 2571 scope.go:117] "RemoveContainer" containerID="aa2366a03d4dfb60b48eeb1c240416307648a2de48687acdc9fd1bc107f30948" Aug 04 00:48:19.965599 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.965577 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg"] Aug 04 00:48:19.975619 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.975600 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4f7c8d-7cff7b98d-cjckg"] Aug 04 00:48:19.984616 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.984596 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv"] Aug 04 00:48:19.991883 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:19.991855 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4f7c8d-5fc57795d6-m2ckv"] Aug 04 00:48:20.002169 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:20.002142 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z"] Aug 04 00:48:20.005260 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:20.005236 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f7c8d-589dfbd5fc-7v89z"] Aug 04 00:48:20.376970 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:20.376936 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c493ee2-61c9-4caa-ab21-9c957e707a22" path="/var/lib/kubelet/pods/1c493ee2-61c9-4caa-ab21-9c957e707a22/volumes" Aug 04 00:48:20.377350 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:20.377335 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" path="/var/lib/kubelet/pods/3158b3c9-6fc0-4bf6-8cc1-408a43766d4e/volumes" Aug 04 00:48:20.377696 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:20.377683 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="553e528a-ab77-4923-a78d-8bb91d495449" path="/var/lib/kubelet/pods/553e528a-ab77-4923-a78d-8bb91d495449/volumes" Aug 04 00:48:20.377972 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:20.377961 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9239ab6b-833c-49f8-867b-a9b0927f17fa" path="/var/lib/kubelet/pods/9239ab6b-833c-49f8-867b-a9b0927f17fa/volumes" Aug 04 00:48:30.768924 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.768888 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt"] Aug 04 00:48:30.769552 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769473 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553e528a-ab77-4923-a78d-8bb91d495449" containerName="maas-api" Aug 04 00:48:30.769552 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769496 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="553e528a-ab77-4923-a78d-8bb91d495449" containerName="maas-api" Aug 04 00:48:30.769552 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769513 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9239ab6b-833c-49f8-867b-a9b0927f17fa" containerName="revoke-keys" Aug 04 00:48:30.769552 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769523 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9239ab6b-833c-49f8-867b-a9b0927f17fa" containerName="revoke-keys" Aug 04 00:48:30.769552 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769550 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" containerName="payload-processing" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769561 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" containerName="payload-processing" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769579 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c493ee2-61c9-4caa-ab21-9c957e707a22" containerName="payload-pre-processing" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769587 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c493ee2-61c9-4caa-ab21-9c957e707a22" containerName="payload-pre-processing" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769681 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="553e528a-ab77-4923-a78d-8bb91d495449" containerName="maas-api" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769693 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c493ee2-61c9-4caa-ab21-9c957e707a22" containerName="payload-pre-processing" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769703 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9239ab6b-833c-49f8-867b-a9b0927f17fa" containerName="revoke-keys" Aug 04 00:48:30.769824 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.769716 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3158b3c9-6fc0-4bf6-8cc1-408a43766d4e" containerName="payload-processing" Aug 04 00:48:30.773154 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.773132 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:30.775373 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.775350 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:48:30.779563 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.779537 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt"] Aug 04 00:48:30.821851 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.821815 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a52ae3-7b43-4134-b788-93162f29b077-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt\" (UID: \"59a52ae3-7b43-4134-b788-93162f29b077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:30.922436 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.922403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a52ae3-7b43-4134-b788-93162f29b077-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt\" (UID: \"59a52ae3-7b43-4134-b788-93162f29b077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:30.923021 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:30.923000 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a52ae3-7b43-4134-b788-93162f29b077-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt\" (UID: \"59a52ae3-7b43-4134-b788-93162f29b077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:31.086935 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:31.086843 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:31.220820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:31.220776 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt"] Aug 04 00:48:31.223793 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:48:31.223762 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a52ae3_7b43_4134_b788_93162f29b077.slice/crio-d1136841a9849d2b58dff319743f10917c6610c13047dadbcbc843877e026d31 WatchSource:0}: Error finding container d1136841a9849d2b58dff319743f10917c6610c13047dadbcbc843877e026d31: Status 404 returned error can't find the container with id d1136841a9849d2b58dff319743f10917c6610c13047dadbcbc843877e026d31 Aug 04 00:48:32.019433 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:32.019394 2571 generic.go:356] "Generic (PLEG): container finished" podID="59a52ae3-7b43-4134-b788-93162f29b077" containerID="55168bd8f3f0658e48f9de857ed72f1ff0186424a13c20b85095e9ed3b92603a" exitCode=0 Aug 04 00:48:32.019433 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:32.019429 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" event={"ID":"59a52ae3-7b43-4134-b788-93162f29b077","Type":"ContainerDied","Data":"55168bd8f3f0658e48f9de857ed72f1ff0186424a13c20b85095e9ed3b92603a"} Aug 04 00:48:32.019857 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:32.019508 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" event={"ID":"59a52ae3-7b43-4134-b788-93162f29b077","Type":"ContainerStarted","Data":"d1136841a9849d2b58dff319743f10917c6610c13047dadbcbc843877e026d31"} Aug 04 00:48:33.163745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:33.163723 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:33.245212 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:33.245177 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a52ae3-7b43-4134-b788-93162f29b077-maas-api-service-ca\") pod \"59a52ae3-7b43-4134-b788-93162f29b077\" (UID: \"59a52ae3-7b43-4134-b788-93162f29b077\") " Aug 04 00:48:33.245521 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:33.245499 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a52ae3-7b43-4134-b788-93162f29b077-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59a52ae3-7b43-4134-b788-93162f29b077" (UID: "59a52ae3-7b43-4134-b788-93162f29b077"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:33.346731 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:33.346640 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a52ae3-7b43-4134-b788-93162f29b077-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:34.030839 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:34.030802 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" Aug 04 00:48:34.030994 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:34.030839 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt" event={"ID":"59a52ae3-7b43-4134-b788-93162f29b077","Type":"ContainerDied","Data":"d1136841a9849d2b58dff319743f10917c6610c13047dadbcbc843877e026d31"} Aug 04 00:48:34.030994 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:34.030886 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1136841a9849d2b58dff319743f10917c6610c13047dadbcbc843877e026d31" Aug 04 00:48:41.179932 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.179898 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh"] Aug 04 00:48:41.180485 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.180185 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" podUID="def84d82-5830-4fac-aad4-b54e9ecb2854" containerName="maas-api" containerID="cri-o://6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5" gracePeriod=30 Aug 04 00:48:41.239743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.239704 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd"] Aug 04 00:48:41.239977 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.239951 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" podUID="a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" containerName="payload-pre-processing" containerID="cri-o://fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438" gracePeriod=30 Aug 04 00:48:41.374598 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.374564 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt"] Aug 04 00:48:41.377865 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.377833 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-743d5d-wtqwt"] Aug 04 00:48:41.495005 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.494979 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:48:41.513248 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.513136 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def84d82-5830-4fac-aad4-b54e9ecb2854-openshift-service-ca\") pod \"def84d82-5830-4fac-aad4-b54e9ecb2854\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " Aug 04 00:48:41.513420 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.513389 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4kwg\" (UniqueName: \"kubernetes.io/projected/def84d82-5830-4fac-aad4-b54e9ecb2854-kube-api-access-r4kwg\") pod \"def84d82-5830-4fac-aad4-b54e9ecb2854\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " Aug 04 00:48:41.516882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.513450 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def84d82-5830-4fac-aad4-b54e9ecb2854-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "def84d82-5830-4fac-aad4-b54e9ecb2854" (UID: "def84d82-5830-4fac-aad4-b54e9ecb2854"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:41.516882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.513503 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls\") pod \"def84d82-5830-4fac-aad4-b54e9ecb2854\" (UID: \"def84d82-5830-4fac-aad4-b54e9ecb2854\") " Aug 04 00:48:41.516882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.513763 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def84d82-5830-4fac-aad4-b54e9ecb2854-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:41.516882 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.516772 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def84d82-5830-4fac-aad4-b54e9ecb2854-kube-api-access-r4kwg" (OuterVolumeSpecName: "kube-api-access-r4kwg") pod "def84d82-5830-4fac-aad4-b54e9ecb2854" (UID: "def84d82-5830-4fac-aad4-b54e9ecb2854"). InnerVolumeSpecName "kube-api-access-r4kwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:41.517250 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.517147 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "def84d82-5830-4fac-aad4-b54e9ecb2854" (UID: "def84d82-5830-4fac-aad4-b54e9ecb2854"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:48:41.524841 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.524819 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:48:41.614334 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.614303 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-plugins-config-volume\") pod \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " Aug 04 00:48:41.614529 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.614407 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vv9r\" (UniqueName: \"kubernetes.io/projected/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-kube-api-access-6vv9r\") pod \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\" (UID: \"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e\") " Aug 04 00:48:41.614658 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.614642 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def84d82-5830-4fac-aad4-b54e9ecb2854-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:41.614658 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.614643 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" (UID: "a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:48:41.614750 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.614663 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kwg\" (UniqueName: \"kubernetes.io/projected/def84d82-5830-4fac-aad4-b54e9ecb2854-kube-api-access-r4kwg\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:41.616354 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.616334 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-kube-api-access-6vv9r" (OuterVolumeSpecName: "kube-api-access-6vv9r") pod "a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" (UID: "a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e"). InnerVolumeSpecName "kube-api-access-6vv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:48:41.715642 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.715606 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vv9r\" (UniqueName: \"kubernetes.io/projected/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-kube-api-access-6vv9r\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:41.715642 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:41.715636 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:48:42.075803 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.075766 2571 generic.go:356] "Generic (PLEG): container finished" podID="def84d82-5830-4fac-aad4-b54e9ecb2854" containerID="6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5" exitCode=0 Aug 04 00:48:42.076002 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.075809 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" Aug 04 00:48:42.076002 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.075858 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" event={"ID":"def84d82-5830-4fac-aad4-b54e9ecb2854","Type":"ContainerDied","Data":"6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5"} Aug 04 00:48:42.076002 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.075907 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh" event={"ID":"def84d82-5830-4fac-aad4-b54e9ecb2854","Type":"ContainerDied","Data":"5e08b09ef7299625d22581412010406c0f45770c8ddb070885b2486f9dcc9461"} Aug 04 00:48:42.076002 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.075937 2571 scope.go:117] "RemoveContainer" containerID="6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5" Aug 04 00:48:42.077384 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.077352 2571 generic.go:356] "Generic (PLEG): container finished" podID="a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" containerID="fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438" exitCode=0 Aug 04 00:48:42.077509 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.077432 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" Aug 04 00:48:42.077509 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.077446 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" event={"ID":"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e","Type":"ContainerDied","Data":"fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438"} Aug 04 00:48:42.077509 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.077501 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd" event={"ID":"a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e","Type":"ContainerDied","Data":"e11a2d4c4c9c40371f8e35bde79f9e0671d7e7e7eb4ce9b197de0211228faf57"} Aug 04 00:48:42.085321 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.085302 2571 scope.go:117] "RemoveContainer" containerID="6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5" Aug 04 00:48:42.085631 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:48:42.085604 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5\": container with ID starting with 6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5 not found: ID does not exist" containerID="6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5" Aug 04 00:48:42.085719 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.085644 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5"} err="failed to get container status \"6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5\": rpc error: code = NotFound desc = could not find container \"6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5\": container with ID starting with 6e85660677631ff2f77f367ecdf3c11990d9e7c8e5bed94ad7ab88062cd7dea5 not found: ID does not exist" Aug 04 00:48:42.085719 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.085670 2571 scope.go:117] "RemoveContainer" containerID="fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438" Aug 04 00:48:42.094592 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.094573 2571 scope.go:117] "RemoveContainer" containerID="fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438" Aug 04 00:48:42.094832 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:48:42.094813 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438\": container with ID starting with fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438 not found: ID does not exist" containerID="fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438" Aug 04 00:48:42.094876 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.094839 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438"} err="failed to get container status \"fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438\": rpc error: code = NotFound desc = could not find container \"fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438\": container with ID starting with fa83ad3f3c1b1ebd1e4b596418e4e3c4f90d3970195009394b3497d14cbf9438 not found: ID does not exist" Aug 04 00:48:42.102045 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.102019 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd"] Aug 04 00:48:42.105602 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.105583 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-743d5d-7ff4746cb4-nppjd"] Aug 04 00:48:42.114920 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.114883 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh"] Aug 04 00:48:42.117629 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.117611 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-743d5d-85974cdc7f-fpsjh"] Aug 04 00:48:42.382300 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.382222 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a52ae3-7b43-4134-b788-93162f29b077" path="/var/lib/kubelet/pods/59a52ae3-7b43-4134-b788-93162f29b077/volumes" Aug 04 00:48:42.382646 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.382603 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" path="/var/lib/kubelet/pods/a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e/volumes" Aug 04 00:48:42.382948 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:42.382934 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def84d82-5830-4fac-aad4-b54e9ecb2854" path="/var/lib/kubelet/pods/def84d82-5830-4fac-aad4-b54e9ecb2854/volumes" Aug 04 00:48:52.716521 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.716488 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m"] Aug 04 00:48:52.717008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.716964 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a52ae3-7b43-4134-b788-93162f29b077" containerName="revoke-keys" Aug 04 00:48:52.717008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.716978 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a52ae3-7b43-4134-b788-93162f29b077" containerName="revoke-keys" Aug 04 00:48:52.717008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.716996 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def84d82-5830-4fac-aad4-b54e9ecb2854" containerName="maas-api" Aug 04 00:48:52.717008 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.717001 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="def84d82-5830-4fac-aad4-b54e9ecb2854" containerName="maas-api" Aug 04 00:48:52.717309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.717018 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" containerName="payload-pre-processing" Aug 04 00:48:52.717309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.717026 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" containerName="payload-pre-processing" Aug 04 00:48:52.717309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.717087 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a965b1f1-f5d0-4df0-b8ed-a45d0eb2992e" containerName="payload-pre-processing" Aug 04 00:48:52.717309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.717096 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="def84d82-5830-4fac-aad4-b54e9ecb2854" containerName="maas-api" Aug 04 00:48:52.717309 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.717123 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a52ae3-7b43-4134-b788-93162f29b077" containerName="revoke-keys" Aug 04 00:48:52.720226 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.720205 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.722322 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.722295 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a12ead-serving-cert\"" Aug 04 00:48:52.731540 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.731512 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m"] Aug 04 00:48:52.805986 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.805948 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqjm\" (UniqueName: \"kubernetes.io/projected/3158779e-656a-4540-96f9-c428562ddd47-kube-api-access-dvqjm\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.806185 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.806058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3158779e-656a-4540-96f9-c428562ddd47-maas-api-tls\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.806185 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.806157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3158779e-656a-4540-96f9-c428562ddd47-openshift-service-ca\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.907447 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.907411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3158779e-656a-4540-96f9-c428562ddd47-openshift-service-ca\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.907447 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.907459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqjm\" (UniqueName: \"kubernetes.io/projected/3158779e-656a-4540-96f9-c428562ddd47-kube-api-access-dvqjm\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.907722 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.907506 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3158779e-656a-4540-96f9-c428562ddd47-maas-api-tls\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.908054 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.908032 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3158779e-656a-4540-96f9-c428562ddd47-openshift-service-ca\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.909924 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.909902 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3158779e-656a-4540-96f9-c428562ddd47-maas-api-tls\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:52.914935 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:52.914914 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqjm\" (UniqueName: \"kubernetes.io/projected/3158779e-656a-4540-96f9-c428562ddd47-kube-api-access-dvqjm\") pod \"maas-api-e2e-api-a-a12ead-69b947db-c5v9m\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:53.032679 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.032646 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:53.170733 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.170700 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m"] Aug 04 00:48:53.172541 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:48:53.172514 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3158779e_656a_4540_96f9_c428562ddd47.slice/crio-8bb16941615b3725b2dd33b3be7354317aa2864060148806f2aa18f5a55634d9 WatchSource:0}: Error finding container 8bb16941615b3725b2dd33b3be7354317aa2864060148806f2aa18f5a55634d9: Status 404 returned error can't find the container with id 8bb16941615b3725b2dd33b3be7354317aa2864060148806f2aa18f5a55634d9 Aug 04 00:48:53.173902 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.173882 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 00:48:53.393019 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.392933 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm"] Aug 04 00:48:53.397515 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.397498 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.401349 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.401326 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a12ead\"" Aug 04 00:48:53.408471 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.408450 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm"] Aug 04 00:48:53.458990 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.458962 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld"] Aug 04 00:48:53.462356 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.462338 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.464741 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.464722 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a12ead-dockercfg-8hmff\"" Aug 04 00:48:53.477924 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.477897 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld"] Aug 04 00:48:53.516663 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.513897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwk7\" (UniqueName: \"kubernetes.io/projected/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-kube-api-access-jwwk7\") pod \"payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.516663 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.514026 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6pg2\" (UniqueName: \"kubernetes.io/projected/62379cd8-aa74-428d-976b-377ecc71e558-kube-api-access-v6pg2\") pod \"payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.516663 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.514074 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.516663 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.514142 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62379cd8-aa74-428d-976b-377ecc71e558-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.615386 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.615350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwk7\" (UniqueName: \"kubernetes.io/projected/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-kube-api-access-jwwk7\") pod \"payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.615588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.615504 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6pg2\" (UniqueName: \"kubernetes.io/projected/62379cd8-aa74-428d-976b-377ecc71e558-kube-api-access-v6pg2\") pod \"payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.615588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.615547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.615588 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.615576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62379cd8-aa74-428d-976b-377ecc71e558-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.616332 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.616307 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62379cd8-aa74-428d-976b-377ecc71e558-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.616435 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.616315 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.622940 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.622905 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwk7\" (UniqueName: \"kubernetes.io/projected/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-kube-api-access-jwwk7\") pod \"payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.623050 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.623026 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6pg2\" (UniqueName: \"kubernetes.io/projected/62379cd8-aa74-428d-976b-377ecc71e558-kube-api-access-v6pg2\") pod \"payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.708517 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.708422 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:53.772813 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.772780 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:53.858966 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.858829 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm"] Aug 04 00:48:53.861235 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:48:53.861208 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31411ad6_2bdd_4b2f_9146_f5f29e9d5fed.slice/crio-a1aff9e802bd029fc0beeaff909cc8e3c11375e490d0e5efe4c0da8430be6e9c WatchSource:0}: Error finding container a1aff9e802bd029fc0beeaff909cc8e3c11375e490d0e5efe4c0da8430be6e9c: Status 404 returned error can't find the container with id a1aff9e802bd029fc0beeaff909cc8e3c11375e490d0e5efe4c0da8430be6e9c Aug 04 00:48:53.933838 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:53.933810 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld"] Aug 04 00:48:53.935890 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:48:53.935862 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62379cd8_aa74_428d_976b_377ecc71e558.slice/crio-c24795c16fe5db9b3bd625c427899e9e6acc130c04513ab4d3286c8529c5b427 WatchSource:0}: Error finding container c24795c16fe5db9b3bd625c427899e9e6acc130c04513ab4d3286c8529c5b427: Status 404 returned error can't find the container with id c24795c16fe5db9b3bd625c427899e9e6acc130c04513ab4d3286c8529c5b427 Aug 04 00:48:54.147873 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.147831 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" event={"ID":"62379cd8-aa74-428d-976b-377ecc71e558","Type":"ContainerStarted","Data":"745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91"} Aug 04 00:48:54.148088 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.147905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" event={"ID":"62379cd8-aa74-428d-976b-377ecc71e558","Type":"ContainerStarted","Data":"c24795c16fe5db9b3bd625c427899e9e6acc130c04513ab4d3286c8529c5b427"} Aug 04 00:48:54.148088 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.148058 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:48:54.149620 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.149591 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" event={"ID":"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed","Type":"ContainerStarted","Data":"89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162"} Aug 04 00:48:54.149755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.149638 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" event={"ID":"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed","Type":"ContainerStarted","Data":"a1aff9e802bd029fc0beeaff909cc8e3c11375e490d0e5efe4c0da8430be6e9c"} Aug 04 00:48:54.149755 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.149670 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:48:54.151443 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.151420 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" event={"ID":"3158779e-656a-4540-96f9-c428562ddd47","Type":"ContainerStarted","Data":"4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27"} Aug 04 00:48:54.151560 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.151460 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" event={"ID":"3158779e-656a-4540-96f9-c428562ddd47","Type":"ContainerStarted","Data":"8bb16941615b3725b2dd33b3be7354317aa2864060148806f2aa18f5a55634d9"} Aug 04 00:48:54.151560 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.151487 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:48:54.168197 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.168151 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" podStartSLOduration=1.1681371839999999 podStartE2EDuration="1.168137184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:48:54.165661964 +0000 UTC m=+2772.336695272" watchObservedRunningTime="2026-08-04 00:48:54.168137184 +0000 UTC m=+2772.339170490" Aug 04 00:48:54.182274 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.182223 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" podStartSLOduration=1.18220554 podStartE2EDuration="1.18220554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:48:54.181067044 +0000 UTC m=+2772.352100354" watchObservedRunningTime="2026-08-04 00:48:54.18220554 +0000 UTC m=+2772.353238850" Aug 04 00:48:54.203042 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:48:54.202989 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" podStartSLOduration=1.577749555 podStartE2EDuration="2.202973774s" totalImagesPullingTime="625.224219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:48:54.199775464 +0000 UTC m=+2772.370808772" watchObservedRunningTime="2026-08-04 00:48:54.202973774 +0000 UTC m=+2772.374007081" Aug 04 00:49:00.163792 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:00.163760 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:49:05.160335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:05.160256 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:49:05.160862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:05.160584 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:49:20.782449 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.782407 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj"] Aug 04 00:49:20.787314 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.787292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:20.789444 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.789421 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5782b0-serving-cert\"" Aug 04 00:49:20.795717 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.795686 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj"] Aug 04 00:49:20.965329 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.965296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b3570-4042-4031-b922-adb629760439-openshift-service-ca\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:20.965512 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.965364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:20.965512 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:20.965430 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpv4\" (UniqueName: \"kubernetes.io/projected/114b3570-4042-4031-b922-adb629760439-kube-api-access-hbpv4\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.066216 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.066127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv4\" (UniqueName: \"kubernetes.io/projected/114b3570-4042-4031-b922-adb629760439-kube-api-access-hbpv4\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.066216 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.066179 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b3570-4042-4031-b922-adb629760439-openshift-service-ca\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.066416 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.066220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.066416 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:49:21.066317 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-serving-cert: secret "maas-api-e2e-api-b-5782b0-serving-cert" not found Aug 04 00:49:21.066416 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:49:21.066369 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls podName:114b3570-4042-4031-b922-adb629760439 nodeName:}" failed. No retries permitted until 2026-08-04 00:49:21.566352584 +0000 UTC m=+2799.737385869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls") pod "maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" (UID: "114b3570-4042-4031-b922-adb629760439") : secret "maas-api-e2e-api-b-5782b0-serving-cert" not found Aug 04 00:49:21.066947 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.066929 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b3570-4042-4031-b922-adb629760439-openshift-service-ca\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.074749 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.074728 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpv4\" (UniqueName: \"kubernetes.io/projected/114b3570-4042-4031-b922-adb629760439-kube-api-access-hbpv4\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.444218 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.444132 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp"] Aug 04 00:49:21.447693 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.447670 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.451335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.451296 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5782b0-dockercfg-8lq7z\"" Aug 04 00:49:21.451335 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.451316 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5782b0\"" Aug 04 00:49:21.460236 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.460216 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp"] Aug 04 00:49:21.572142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.572083 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.572349 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.572165 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29c9\" (UniqueName: \"kubernetes.io/projected/5535d877-6165-4806-bb66-2b2f97c8844b-kube-api-access-l29c9\") pod \"payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.572349 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.572205 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535d877-6165-4806-bb66-2b2f97c8844b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.575248 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.575210 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls\") pod \"maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.673405 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.673373 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29c9\" (UniqueName: \"kubernetes.io/projected/5535d877-6165-4806-bb66-2b2f97c8844b-kube-api-access-l29c9\") pod \"payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.673586 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.673427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535d877-6165-4806-bb66-2b2f97c8844b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.674165 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.674146 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535d877-6165-4806-bb66-2b2f97c8844b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.680978 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.680948 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29c9\" (UniqueName: \"kubernetes.io/projected/5535d877-6165-4806-bb66-2b2f97c8844b-kube-api-access-l29c9\") pod \"payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.707150 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.707076 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:21.759147 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.759088 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:21.856139 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.856083 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj"] Aug 04 00:49:21.857378 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:49:21.857326 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114b3570_4042_4031_b922_adb629760439.slice/crio-59e8bdb1919fca867166819cf14469b3f35282a9e4f0faaa28673dc80c7162c9 WatchSource:0}: Error finding container 59e8bdb1919fca867166819cf14469b3f35282a9e4f0faaa28673dc80c7162c9: Status 404 returned error can't find the container with id 59e8bdb1919fca867166819cf14469b3f35282a9e4f0faaa28673dc80c7162c9 Aug 04 00:49:21.909490 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:21.909460 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp"] Aug 04 00:49:21.911045 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:49:21.911015 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5535d877_6165_4806_bb66_2b2f97c8844b.slice/crio-e7e5c5b034a0c2076ad53ef4294f59244d433023e02f9d9d880cf42b6d462aa1 WatchSource:0}: Error finding container e7e5c5b034a0c2076ad53ef4294f59244d433023e02f9d9d880cf42b6d462aa1: Status 404 returned error can't find the container with id e7e5c5b034a0c2076ad53ef4294f59244d433023e02f9d9d880cf42b6d462aa1 Aug 04 00:49:22.314857 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:22.314824 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" event={"ID":"5535d877-6165-4806-bb66-2b2f97c8844b","Type":"ContainerStarted","Data":"9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1"} Aug 04 00:49:22.315064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:22.314885 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" event={"ID":"5535d877-6165-4806-bb66-2b2f97c8844b","Type":"ContainerStarted","Data":"e7e5c5b034a0c2076ad53ef4294f59244d433023e02f9d9d880cf42b6d462aa1"} Aug 04 00:49:22.315064 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:22.314932 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:22.316547 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:22.316521 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" event={"ID":"114b3570-4042-4031-b922-adb629760439","Type":"ContainerStarted","Data":"59e8bdb1919fca867166819cf14469b3f35282a9e4f0faaa28673dc80c7162c9"} Aug 04 00:49:22.354170 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:22.354118 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" podStartSLOduration=1.354089165 podStartE2EDuration="1.354089165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:49:22.352218148 +0000 UTC m=+2800.523251468" watchObservedRunningTime="2026-08-04 00:49:22.354089165 +0000 UTC m=+2800.525122471" Aug 04 00:49:23.324209 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:23.324171 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" event={"ID":"114b3570-4042-4031-b922-adb629760439","Type":"ContainerStarted","Data":"b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d"} Aug 04 00:49:23.324642 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:23.324252 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:23.339217 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:23.339157 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" podStartSLOduration=2.700027166 podStartE2EDuration="3.339137542s" totalImagesPullingTime="639.110376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:49:23.337910481 +0000 UTC m=+2801.508943788" watchObservedRunningTime="2026-08-04 00:49:23.339137542 +0000 UTC m=+2801.510170849" Aug 04 00:49:29.335542 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:29.335508 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:49:33.327279 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:33.327250 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:49:47.397491 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.397451 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs"] Aug 04 00:49:47.401478 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.401456 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:47.403908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.403869 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:49:47.407511 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.407489 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs"] Aug 04 00:49:47.503594 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.503560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a52201-5336-4bd7-bf50-358e2aec1bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs\" (UID: \"e2a52201-5336-4bd7-bf50-358e2aec1bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:47.604387 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.604349 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a52201-5336-4bd7-bf50-358e2aec1bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs\" (UID: \"e2a52201-5336-4bd7-bf50-358e2aec1bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:47.605032 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.605005 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a52201-5336-4bd7-bf50-358e2aec1bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs\" (UID: \"e2a52201-5336-4bd7-bf50-358e2aec1bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:47.713669 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.713575 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:47.849756 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:47.849728 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs"] Aug 04 00:49:47.850654 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:49:47.850626 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a52201_5336_4bd7_bf50_358e2aec1bdf.slice/crio-9b82aaaeddc7382b4bfca859c4c4151a2173faf31a298691fa89c84b2f9a2f18 WatchSource:0}: Error finding container 9b82aaaeddc7382b4bfca859c4c4151a2173faf31a298691fa89c84b2f9a2f18: Status 404 returned error can't find the container with id 9b82aaaeddc7382b4bfca859c4c4151a2173faf31a298691fa89c84b2f9a2f18 Aug 04 00:49:48.471198 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:48.471154 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" event={"ID":"e2a52201-5336-4bd7-bf50-358e2aec1bdf","Type":"ContainerStarted","Data":"945049c67ee177dfe1d3600a3dda0964f520d84db5e55b183a96555b24dcfd8d"} Aug 04 00:49:48.471638 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:48.471234 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" event={"ID":"e2a52201-5336-4bd7-bf50-358e2aec1bdf","Type":"ContainerStarted","Data":"9b82aaaeddc7382b4bfca859c4c4151a2173faf31a298691fa89c84b2f9a2f18"} Aug 04 00:49:48.486795 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:48.486739 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" podStartSLOduration=1.486725025 podStartE2EDuration="1.486725025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:49:48.485586737 +0000 UTC m=+2826.656620044" watchObservedRunningTime="2026-08-04 00:49:48.486725025 +0000 UTC m=+2826.657758392" Aug 04 00:49:49.478766 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:49.478730 2571 generic.go:356] "Generic (PLEG): container finished" podID="e2a52201-5336-4bd7-bf50-358e2aec1bdf" containerID="945049c67ee177dfe1d3600a3dda0964f520d84db5e55b183a96555b24dcfd8d" exitCode=0 Aug 04 00:49:49.479200 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:49.478817 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" event={"ID":"e2a52201-5336-4bd7-bf50-358e2aec1bdf","Type":"ContainerDied","Data":"945049c67ee177dfe1d3600a3dda0964f520d84db5e55b183a96555b24dcfd8d"} Aug 04 00:49:50.614279 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:50.614248 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:50.735589 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:50.735498 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a52201-5336-4bd7-bf50-358e2aec1bdf-maas-api-service-ca\") pod \"e2a52201-5336-4bd7-bf50-358e2aec1bdf\" (UID: \"e2a52201-5336-4bd7-bf50-358e2aec1bdf\") " Aug 04 00:49:50.735878 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:50.735853 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a52201-5336-4bd7-bf50-358e2aec1bdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2a52201-5336-4bd7-bf50-358e2aec1bdf" (UID: "e2a52201-5336-4bd7-bf50-358e2aec1bdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:49:50.836970 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:50.836936 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a52201-5336-4bd7-bf50-358e2aec1bdf-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:49:51.491591 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:51.491557 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" Aug 04 00:49:51.491763 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:51.491594 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs" event={"ID":"e2a52201-5336-4bd7-bf50-358e2aec1bdf","Type":"ContainerDied","Data":"9b82aaaeddc7382b4bfca859c4c4151a2173faf31a298691fa89c84b2f9a2f18"} Aug 04 00:49:51.491763 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:49:51.491644 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b82aaaeddc7382b4bfca859c4c4151a2173faf31a298691fa89c84b2f9a2f18" Aug 04 00:50:02.913272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:02.913234 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m"] Aug 04 00:50:02.913772 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:02.913493 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" podUID="3158779e-656a-4540-96f9-c428562ddd47" containerName="maas-api" containerID="cri-o://4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27" gracePeriod=30 Aug 04 00:50:02.971047 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:02.971007 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm"] Aug 04 00:50:02.971365 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:02.971307 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" podUID="31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" containerName="payload-processing" containerID="cri-o://89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162" gracePeriod=30 Aug 04 00:50:02.977132 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:02.977081 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld"] Aug 04 00:50:02.977539 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:02.977508 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" podUID="62379cd8-aa74-428d-976b-377ecc71e558" containerName="payload-pre-processing" containerID="cri-o://745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91" gracePeriod=30 Aug 04 00:50:03.123379 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.123332 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs"] Aug 04 00:50:03.127123 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.127066 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a12ead-8xhjs"] Aug 04 00:50:03.281691 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.281666 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:50:03.284892 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.284872 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:50:03.288285 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.288266 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:50:03.448644 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448546 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-plugins-config-volume\") pod \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " Aug 04 00:50:03.448644 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448607 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwwk7\" (UniqueName: \"kubernetes.io/projected/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-kube-api-access-jwwk7\") pod \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\" (UID: \"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed\") " Aug 04 00:50:03.448879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448652 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3158779e-656a-4540-96f9-c428562ddd47-maas-api-tls\") pod \"3158779e-656a-4540-96f9-c428562ddd47\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " Aug 04 00:50:03.448879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448675 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3158779e-656a-4540-96f9-c428562ddd47-openshift-service-ca\") pod \"3158779e-656a-4540-96f9-c428562ddd47\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " Aug 04 00:50:03.448879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448716 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62379cd8-aa74-428d-976b-377ecc71e558-plugins-config-volume\") pod \"62379cd8-aa74-428d-976b-377ecc71e558\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " Aug 04 00:50:03.448879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448742 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6pg2\" (UniqueName: \"kubernetes.io/projected/62379cd8-aa74-428d-976b-377ecc71e558-kube-api-access-v6pg2\") pod \"62379cd8-aa74-428d-976b-377ecc71e558\" (UID: \"62379cd8-aa74-428d-976b-377ecc71e558\") " Aug 04 00:50:03.448879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.448780 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvqjm\" (UniqueName: \"kubernetes.io/projected/3158779e-656a-4540-96f9-c428562ddd47-kube-api-access-dvqjm\") pod \"3158779e-656a-4540-96f9-c428562ddd47\" (UID: \"3158779e-656a-4540-96f9-c428562ddd47\") " Aug 04 00:50:03.449175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.449038 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" (UID: "31411ad6-2bdd-4b2f-9146-f5f29e9d5fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:03.449175 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.449038 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3158779e-656a-4540-96f9-c428562ddd47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3158779e-656a-4540-96f9-c428562ddd47" (UID: "3158779e-656a-4540-96f9-c428562ddd47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:03.449299 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.449213 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62379cd8-aa74-428d-976b-377ecc71e558-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62379cd8-aa74-428d-976b-377ecc71e558" (UID: "62379cd8-aa74-428d-976b-377ecc71e558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:03.450950 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.450923 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3158779e-656a-4540-96f9-c428562ddd47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3158779e-656a-4540-96f9-c428562ddd47" (UID: "3158779e-656a-4540-96f9-c428562ddd47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:50:03.450950 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.450944 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-kube-api-access-jwwk7" (OuterVolumeSpecName: "kube-api-access-jwwk7") pod "31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" (UID: "31411ad6-2bdd-4b2f-9146-f5f29e9d5fed"). InnerVolumeSpecName "kube-api-access-jwwk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:03.451302 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.451270 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62379cd8-aa74-428d-976b-377ecc71e558-kube-api-access-v6pg2" (OuterVolumeSpecName: "kube-api-access-v6pg2") pod "62379cd8-aa74-428d-976b-377ecc71e558" (UID: "62379cd8-aa74-428d-976b-377ecc71e558"). InnerVolumeSpecName "kube-api-access-v6pg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:03.451389 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.451279 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3158779e-656a-4540-96f9-c428562ddd47-kube-api-access-dvqjm" (OuterVolumeSpecName: "kube-api-access-dvqjm") pod "3158779e-656a-4540-96f9-c428562ddd47" (UID: "3158779e-656a-4540-96f9-c428562ddd47"). InnerVolumeSpecName "kube-api-access-dvqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:03.549946 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.549901 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62379cd8-aa74-428d-976b-377ecc71e558-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.549946 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.549938 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6pg2\" (UniqueName: \"kubernetes.io/projected/62379cd8-aa74-428d-976b-377ecc71e558-kube-api-access-v6pg2\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.549946 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.549951 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqjm\" (UniqueName: \"kubernetes.io/projected/3158779e-656a-4540-96f9-c428562ddd47-kube-api-access-dvqjm\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.550247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.549966 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.550247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.549977 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwk7\" (UniqueName: \"kubernetes.io/projected/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed-kube-api-access-jwwk7\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.550247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.549989 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3158779e-656a-4540-96f9-c428562ddd47-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.550247 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.550002 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3158779e-656a-4540-96f9-c428562ddd47-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:03.557927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.557893 2571 generic.go:356] "Generic (PLEG): container finished" podID="62379cd8-aa74-428d-976b-377ecc71e558" containerID="745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91" exitCode=0 Aug 04 00:50:03.558072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.557943 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" Aug 04 00:50:03.558072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.557976 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" event={"ID":"62379cd8-aa74-428d-976b-377ecc71e558","Type":"ContainerDied","Data":"745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91"} Aug 04 00:50:03.558072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.558031 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld" event={"ID":"62379cd8-aa74-428d-976b-377ecc71e558","Type":"ContainerDied","Data":"c24795c16fe5db9b3bd625c427899e9e6acc130c04513ab4d3286c8529c5b427"} Aug 04 00:50:03.558072 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.558055 2571 scope.go:117] "RemoveContainer" containerID="745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91" Aug 04 00:50:03.559294 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.559272 2571 generic.go:356] "Generic (PLEG): container finished" podID="3158779e-656a-4540-96f9-c428562ddd47" containerID="4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27" exitCode=0 Aug 04 00:50:03.559402 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.559299 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" event={"ID":"3158779e-656a-4540-96f9-c428562ddd47","Type":"ContainerDied","Data":"4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27"} Aug 04 00:50:03.559402 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.559317 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" Aug 04 00:50:03.559402 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.559358 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m" event={"ID":"3158779e-656a-4540-96f9-c428562ddd47","Type":"ContainerDied","Data":"8bb16941615b3725b2dd33b3be7354317aa2864060148806f2aa18f5a55634d9"} Aug 04 00:50:03.560927 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.560905 2571 generic.go:356] "Generic (PLEG): container finished" podID="31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" containerID="89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162" exitCode=0 Aug 04 00:50:03.561032 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.560949 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" event={"ID":"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed","Type":"ContainerDied","Data":"89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162"} Aug 04 00:50:03.561032 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.560962 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" Aug 04 00:50:03.561032 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.560988 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm" event={"ID":"31411ad6-2bdd-4b2f-9146-f5f29e9d5fed","Type":"ContainerDied","Data":"a1aff9e802bd029fc0beeaff909cc8e3c11375e490d0e5efe4c0da8430be6e9c"} Aug 04 00:50:03.568665 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.568645 2571 scope.go:117] "RemoveContainer" containerID="745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91" Aug 04 00:50:03.568918 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:03.568895 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91\": container with ID starting with 745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91 not found: ID does not exist" containerID="745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91" Aug 04 00:50:03.568980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.568925 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91"} err="failed to get container status \"745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91\": rpc error: code = NotFound desc = could not find container \"745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91\": container with ID starting with 745209000c346975dac7dd8eb3723e0a4d8c685eabb769c2c661ab4077919d91 not found: ID does not exist" Aug 04 00:50:03.568980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.568941 2571 scope.go:117] "RemoveContainer" containerID="4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27" Aug 04 00:50:03.578278 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.578259 2571 scope.go:117] "RemoveContainer" containerID="4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27" Aug 04 00:50:03.578542 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:03.578524 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27\": container with ID starting with 4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27 not found: ID does not exist" containerID="4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27" Aug 04 00:50:03.578589 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.578549 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27"} err="failed to get container status \"4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27\": rpc error: code = NotFound desc = could not find container \"4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27\": container with ID starting with 4f6947561d88b110d87d588bdce41fef54a3bd3b5ba057c8018d7cdf9f818d27 not found: ID does not exist" Aug 04 00:50:03.578589 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.578568 2571 scope.go:117] "RemoveContainer" containerID="89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162" Aug 04 00:50:03.580591 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.580563 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld"] Aug 04 00:50:03.583562 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.583541 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a12ead-75d859949b-4p5ld"] Aug 04 00:50:03.587655 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.587640 2571 scope.go:117] "RemoveContainer" containerID="89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162" Aug 04 00:50:03.587903 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:03.587884 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162\": container with ID starting with 89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162 not found: ID does not exist" containerID="89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162" Aug 04 00:50:03.587954 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.587910 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162"} err="failed to get container status \"89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162\": rpc error: code = NotFound desc = could not find container \"89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162\": container with ID starting with 89826e9e818e177f4835789a3c47514753d8b4bba5b607cda47526ee2b736162 not found: ID does not exist" Aug 04 00:50:03.592713 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.592689 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m"] Aug 04 00:50:03.594825 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.594806 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a12ead-69b947db-c5v9m"] Aug 04 00:50:03.602630 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.602607 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm"] Aug 04 00:50:03.605467 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:03.605444 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a12ead-5d89475f7b-s45sm"] Aug 04 00:50:04.377015 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:04.376982 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" path="/var/lib/kubelet/pods/31411ad6-2bdd-4b2f-9146-f5f29e9d5fed/volumes" Aug 04 00:50:04.377440 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:04.377389 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3158779e-656a-4540-96f9-c428562ddd47" path="/var/lib/kubelet/pods/3158779e-656a-4540-96f9-c428562ddd47/volumes" Aug 04 00:50:04.377706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:04.377692 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62379cd8-aa74-428d-976b-377ecc71e558" path="/var/lib/kubelet/pods/62379cd8-aa74-428d-976b-377ecc71e558/volumes" Aug 04 00:50:04.377977 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:04.377967 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a52201-5336-4bd7-bf50-358e2aec1bdf" path="/var/lib/kubelet/pods/e2a52201-5336-4bd7-bf50-358e2aec1bdf/volumes" Aug 04 00:50:09.831980 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.831942 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87"] Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832323 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" containerName="payload-processing" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832334 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" containerName="payload-processing" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832347 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3158779e-656a-4540-96f9-c428562ddd47" containerName="maas-api" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832354 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3158779e-656a-4540-96f9-c428562ddd47" containerName="maas-api" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832362 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a52201-5336-4bd7-bf50-358e2aec1bdf" containerName="revoke-keys" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832368 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a52201-5336-4bd7-bf50-358e2aec1bdf" containerName="revoke-keys" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832375 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62379cd8-aa74-428d-976b-377ecc71e558" containerName="payload-pre-processing" Aug 04 00:50:09.832403 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832381 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="62379cd8-aa74-428d-976b-377ecc71e558" containerName="payload-pre-processing" Aug 04 00:50:09.832716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832456 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="31411ad6-2bdd-4b2f-9146-f5f29e9d5fed" containerName="payload-processing" Aug 04 00:50:09.832716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832464 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a52201-5336-4bd7-bf50-358e2aec1bdf" containerName="revoke-keys" Aug 04 00:50:09.832716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832471 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="62379cd8-aa74-428d-976b-377ecc71e558" containerName="payload-pre-processing" Aug 04 00:50:09.832716 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.832478 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3158779e-656a-4540-96f9-c428562ddd47" containerName="maas-api" Aug 04 00:50:09.837355 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.837331 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:09.839690 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.839662 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 00:50:09.843721 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.843698 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87"] Aug 04 00:50:09.910338 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:09.910299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ac52b8-6b4f-48d6-a9ca-23c397192e68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5782b0-65k87\" (UID: \"41ac52b8-6b4f-48d6-a9ca-23c397192e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:10.011166 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:10.011122 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ac52b8-6b4f-48d6-a9ca-23c397192e68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5782b0-65k87\" (UID: \"41ac52b8-6b4f-48d6-a9ca-23c397192e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:10.011680 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:10.011659 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ac52b8-6b4f-48d6-a9ca-23c397192e68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5782b0-65k87\" (UID: \"41ac52b8-6b4f-48d6-a9ca-23c397192e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:10.149879 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:10.149793 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:10.284566 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:10.284538 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87"] Aug 04 00:50:10.286629 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:50:10.286601 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ac52b8_6b4f_48d6_a9ca_23c397192e68.slice/crio-c26ec1d9195faee6142a85ba45baf467ccc2d7eee42e173cbb537854949352f6 WatchSource:0}: Error finding container c26ec1d9195faee6142a85ba45baf467ccc2d7eee42e173cbb537854949352f6: Status 404 returned error can't find the container with id c26ec1d9195faee6142a85ba45baf467ccc2d7eee42e173cbb537854949352f6 Aug 04 00:50:10.606971 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:10.606939 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" event={"ID":"41ac52b8-6b4f-48d6-a9ca-23c397192e68","Type":"ContainerStarted","Data":"c26ec1d9195faee6142a85ba45baf467ccc2d7eee42e173cbb537854949352f6"} Aug 04 00:50:11.613727 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:11.613691 2571 generic.go:356] "Generic (PLEG): container finished" podID="41ac52b8-6b4f-48d6-a9ca-23c397192e68" containerID="8e622e2eff010f201098c0ceb79f35b41e6ecc6b06c194e71c21a080c0075978" exitCode=0 Aug 04 00:50:11.614147 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:11.613780 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" event={"ID":"41ac52b8-6b4f-48d6-a9ca-23c397192e68","Type":"ContainerDied","Data":"8e622e2eff010f201098c0ceb79f35b41e6ecc6b06c194e71c21a080c0075978"} Aug 04 00:50:12.750304 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:12.750277 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:12.831852 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:12.831814 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ac52b8-6b4f-48d6-a9ca-23c397192e68-maas-api-service-ca\") pod \"41ac52b8-6b4f-48d6-a9ca-23c397192e68\" (UID: \"41ac52b8-6b4f-48d6-a9ca-23c397192e68\") " Aug 04 00:50:12.832183 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:12.832159 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ac52b8-6b4f-48d6-a9ca-23c397192e68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41ac52b8-6b4f-48d6-a9ca-23c397192e68" (UID: "41ac52b8-6b4f-48d6-a9ca-23c397192e68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:12.932722 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:12.932640 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ac52b8-6b4f-48d6-a9ca-23c397192e68-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:13.627398 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:13.627362 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" Aug 04 00:50:13.627398 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:13.627376 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87" event={"ID":"41ac52b8-6b4f-48d6-a9ca-23c397192e68","Type":"ContainerDied","Data":"c26ec1d9195faee6142a85ba45baf467ccc2d7eee42e173cbb537854949352f6"} Aug 04 00:50:13.627610 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:13.627426 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26ec1d9195faee6142a85ba45baf467ccc2d7eee42e173cbb537854949352f6" Aug 04 00:50:25.280568 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.280534 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj"] Aug 04 00:50:25.281156 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.280744 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" podUID="114b3570-4042-4031-b922-adb629760439" containerName="maas-api" containerID="cri-o://b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d" gracePeriod=30 Aug 04 00:50:25.335030 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.334998 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp"] Aug 04 00:50:25.335235 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.335213 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" podUID="5535d877-6165-4806-bb66-2b2f97c8844b" containerName="payload-processing" containerID="cri-o://9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1" gracePeriod=30 Aug 04 00:50:25.571301 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.568282 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87"] Aug 04 00:50:25.575662 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.575629 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5782b0-65k87"] Aug 04 00:50:25.598535 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.598506 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:50:25.624614 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.624587 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:50:25.646683 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.646652 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b3570-4042-4031-b922-adb629760439-openshift-service-ca\") pod \"114b3570-4042-4031-b922-adb629760439\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " Aug 04 00:50:25.646821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.646752 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls\") pod \"114b3570-4042-4031-b922-adb629760439\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " Aug 04 00:50:25.646821 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.646812 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbpv4\" (UniqueName: \"kubernetes.io/projected/114b3570-4042-4031-b922-adb629760439-kube-api-access-hbpv4\") pod \"114b3570-4042-4031-b922-adb629760439\" (UID: \"114b3570-4042-4031-b922-adb629760439\") " Aug 04 00:50:25.647096 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.647067 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114b3570-4042-4031-b922-adb629760439-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "114b3570-4042-4031-b922-adb629760439" (UID: "114b3570-4042-4031-b922-adb629760439"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:25.648964 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.648942 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "114b3570-4042-4031-b922-adb629760439" (UID: "114b3570-4042-4031-b922-adb629760439"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 00:50:25.649048 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.648982 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114b3570-4042-4031-b922-adb629760439-kube-api-access-hbpv4" (OuterVolumeSpecName: "kube-api-access-hbpv4") pod "114b3570-4042-4031-b922-adb629760439" (UID: "114b3570-4042-4031-b922-adb629760439"). InnerVolumeSpecName "kube-api-access-hbpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:25.697489 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.697458 2571 generic.go:356] "Generic (PLEG): container finished" podID="5535d877-6165-4806-bb66-2b2f97c8844b" containerID="9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1" exitCode=0 Aug 04 00:50:25.697592 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.697505 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" Aug 04 00:50:25.697592 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.697542 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" event={"ID":"5535d877-6165-4806-bb66-2b2f97c8844b","Type":"ContainerDied","Data":"9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1"} Aug 04 00:50:25.697673 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.697620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp" event={"ID":"5535d877-6165-4806-bb66-2b2f97c8844b","Type":"ContainerDied","Data":"e7e5c5b034a0c2076ad53ef4294f59244d433023e02f9d9d880cf42b6d462aa1"} Aug 04 00:50:25.697673 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.697643 2571 scope.go:117] "RemoveContainer" containerID="9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1" Aug 04 00:50:25.698810 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.698791 2571 generic.go:356] "Generic (PLEG): container finished" podID="114b3570-4042-4031-b922-adb629760439" containerID="b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d" exitCode=0 Aug 04 00:50:25.698888 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.698836 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" Aug 04 00:50:25.698930 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.698880 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" event={"ID":"114b3570-4042-4031-b922-adb629760439","Type":"ContainerDied","Data":"b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d"} Aug 04 00:50:25.698966 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.698925 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj" event={"ID":"114b3570-4042-4031-b922-adb629760439","Type":"ContainerDied","Data":"59e8bdb1919fca867166819cf14469b3f35282a9e4f0faaa28673dc80c7162c9"} Aug 04 00:50:25.707549 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.707530 2571 scope.go:117] "RemoveContainer" containerID="9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1" Aug 04 00:50:25.707797 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:25.707777 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1\": container with ID starting with 9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1 not found: ID does not exist" containerID="9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1" Aug 04 00:50:25.707862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.707803 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1"} err="failed to get container status \"9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1\": rpc error: code = NotFound desc = could not find container \"9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1\": container with ID starting with 9afc74f1ced7709842606b82ee16a38e4936beed356b494883e33997ae38dea1 not found: ID does not exist" Aug 04 00:50:25.707862 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.707820 2571 scope.go:117] "RemoveContainer" containerID="b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d" Aug 04 00:50:25.716010 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.715995 2571 scope.go:117] "RemoveContainer" containerID="b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d" Aug 04 00:50:25.716286 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:25.716267 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d\": container with ID starting with b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d not found: ID does not exist" containerID="b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d" Aug 04 00:50:25.716363 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.716297 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d"} err="failed to get container status \"b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d\": rpc error: code = NotFound desc = could not find container \"b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d\": container with ID starting with b2b3c51e3941c940377fc9d0c6dfcadbfaebf8417f0487325f39f8e493245e5d not found: ID does not exist" Aug 04 00:50:25.720852 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.720828 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj"] Aug 04 00:50:25.724533 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.724512 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5782b0-56b757fdf5-66jnj"] Aug 04 00:50:25.747332 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.747310 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l29c9\" (UniqueName: \"kubernetes.io/projected/5535d877-6165-4806-bb66-2b2f97c8844b-kube-api-access-l29c9\") pod \"5535d877-6165-4806-bb66-2b2f97c8844b\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " Aug 04 00:50:25.747427 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.747401 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535d877-6165-4806-bb66-2b2f97c8844b-plugins-config-volume\") pod \"5535d877-6165-4806-bb66-2b2f97c8844b\" (UID: \"5535d877-6165-4806-bb66-2b2f97c8844b\") " Aug 04 00:50:25.747675 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.747652 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b3570-4042-4031-b922-adb629760439-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:25.747753 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.747678 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/114b3570-4042-4031-b922-adb629760439-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:25.747753 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.747691 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbpv4\" (UniqueName: \"kubernetes.io/projected/114b3570-4042-4031-b922-adb629760439-kube-api-access-hbpv4\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:25.747820 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.747784 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5535d877-6165-4806-bb66-2b2f97c8844b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5535d877-6165-4806-bb66-2b2f97c8844b" (UID: "5535d877-6165-4806-bb66-2b2f97c8844b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 00:50:25.749589 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.749568 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5535d877-6165-4806-bb66-2b2f97c8844b-kube-api-access-l29c9" (OuterVolumeSpecName: "kube-api-access-l29c9") pod "5535d877-6165-4806-bb66-2b2f97c8844b" (UID: "5535d877-6165-4806-bb66-2b2f97c8844b"). InnerVolumeSpecName "kube-api-access-l29c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 00:50:25.848908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.848831 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535d877-6165-4806-bb66-2b2f97c8844b-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:25.848908 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:25.848859 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29c9\" (UniqueName: \"kubernetes.io/projected/5535d877-6165-4806-bb66-2b2f97c8844b-kube-api-access-l29c9\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 00:50:26.020204 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:26.020166 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp"] Aug 04 00:50:26.023849 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:26.023820 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5782b0-7bd49c48f4-wjpnp"] Aug 04 00:50:26.376726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:26.376693 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114b3570-4042-4031-b922-adb629760439" path="/var/lib/kubelet/pods/114b3570-4042-4031-b922-adb629760439/volumes" Aug 04 00:50:26.377142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:26.377021 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ac52b8-6b4f-48d6-a9ca-23c397192e68" path="/var/lib/kubelet/pods/41ac52b8-6b4f-48d6-a9ca-23c397192e68/volumes" Aug 04 00:50:26.377390 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:26.377375 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5535d877-6165-4806-bb66-2b2f97c8844b" path="/var/lib/kubelet/pods/5535d877-6165-4806-bb66-2b2f97c8844b/volumes" Aug 04 00:50:37.181389 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181304 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz"] Aug 04 00:50:37.181745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181691 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5535d877-6165-4806-bb66-2b2f97c8844b" containerName="payload-processing" Aug 04 00:50:37.181745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181703 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5535d877-6165-4806-bb66-2b2f97c8844b" containerName="payload-processing" Aug 04 00:50:37.181745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181718 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114b3570-4042-4031-b922-adb629760439" containerName="maas-api" Aug 04 00:50:37.181745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181724 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="114b3570-4042-4031-b922-adb629760439" containerName="maas-api" Aug 04 00:50:37.181745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181731 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ac52b8-6b4f-48d6-a9ca-23c397192e68" containerName="revoke-keys" Aug 04 00:50:37.181745 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181737 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ac52b8-6b4f-48d6-a9ca-23c397192e68" containerName="revoke-keys" Aug 04 00:50:37.181942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181801 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="114b3570-4042-4031-b922-adb629760439" containerName="maas-api" Aug 04 00:50:37.181942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181809 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ac52b8-6b4f-48d6-a9ca-23c397192e68" containerName="revoke-keys" Aug 04 00:50:37.181942 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.181815 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5535d877-6165-4806-bb66-2b2f97c8844b" containerName="payload-processing" Aug 04 00:50:37.186185 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.186160 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.188506 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.188480 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5ddf6e-serving-cert\"" Aug 04 00:50:37.196563 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.196445 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz"] Aug 04 00:50:37.251109 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.251077 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.251321 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.251160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72lh\" (UniqueName: \"kubernetes.io/projected/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-kube-api-access-t72lh\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.251321 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.251191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.352182 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.352148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72lh\" (UniqueName: \"kubernetes.io/projected/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-kube-api-access-t72lh\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.352393 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.352210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.352393 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.352321 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.352971 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.352946 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.354769 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.354749 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.359950 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.359925 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72lh\" (UniqueName: \"kubernetes.io/projected/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-kube-api-access-t72lh\") pod \"maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.500272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.500182 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:37.640212 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.640179 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz"] Aug 04 00:50:37.641722 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:50:37.641693 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34cc9e4_2c9a_4be3_b3ea_89f5ad3c6abb.slice/crio-b1893f1eb93d229dcd02049ef9efc25f38ab97d318cf0bbba45155066ef7427d WatchSource:0}: Error finding container b1893f1eb93d229dcd02049ef9efc25f38ab97d318cf0bbba45155066ef7427d: Status 404 returned error can't find the container with id b1893f1eb93d229dcd02049ef9efc25f38ab97d318cf0bbba45155066ef7427d Aug 04 00:50:37.768655 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.768621 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" event={"ID":"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb","Type":"ContainerStarted","Data":"b1893f1eb93d229dcd02049ef9efc25f38ab97d318cf0bbba45155066ef7427d"} Aug 04 00:50:37.861186 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.861153 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96"] Aug 04 00:50:37.865872 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.865855 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:37.868570 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.868546 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-5ddf6e\"" Aug 04 00:50:37.868676 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.868636 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-5ddf6e-dockercfg-74gfh\"" Aug 04 00:50:37.876674 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.876653 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96"] Aug 04 00:50:37.898954 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.898926 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt"] Aug 04 00:50:37.902476 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.902460 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:37.914026 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.913949 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt"] Aug 04 00:50:37.957382 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.957335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b091241a-3875-47f5-a6ba-958e47da69a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:37.957563 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.957503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nj5\" (UniqueName: \"kubernetes.io/projected/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-kube-api-access-42nj5\") pod \"payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:37.957627 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.957556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzcc\" (UniqueName: \"kubernetes.io/projected/b091241a-3875-47f5-a6ba-958e47da69a0-kube-api-access-9fzcc\") pod \"payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:37.957627 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:37.957596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.058460 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.058429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nj5\" (UniqueName: \"kubernetes.io/projected/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-kube-api-access-42nj5\") pod \"payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.058567 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.058481 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzcc\" (UniqueName: \"kubernetes.io/projected/b091241a-3875-47f5-a6ba-958e47da69a0-kube-api-access-9fzcc\") pod \"payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:38.058640 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.058548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.058702 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.058640 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b091241a-3875-47f5-a6ba-958e47da69a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:38.059362 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.059341 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.059475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.059456 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b091241a-3875-47f5-a6ba-958e47da69a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:38.065487 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.065458 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzcc\" (UniqueName: \"kubernetes.io/projected/b091241a-3875-47f5-a6ba-958e47da69a0-kube-api-access-9fzcc\") pod \"payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:38.065597 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.065575 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nj5\" (UniqueName: \"kubernetes.io/projected/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-kube-api-access-42nj5\") pod \"payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.178990 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.178952 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.213096 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.213064 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:38.340434 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.340410 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96"] Aug 04 00:50:38.343308 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:50:38.343282 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fe2a82_f6ba_4c6b_b5bb_624fa5337d97.slice/crio-96a3882b2bf84ed3e001ddde1e4bad3e342654f4b864ac977da431b552505761 WatchSource:0}: Error finding container 96a3882b2bf84ed3e001ddde1e4bad3e342654f4b864ac977da431b552505761: Status 404 returned error can't find the container with id 96a3882b2bf84ed3e001ddde1e4bad3e342654f4b864ac977da431b552505761 Aug 04 00:50:38.363062 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.362999 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt"] Aug 04 00:50:38.365055 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:50:38.365013 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb091241a_3875_47f5_a6ba_958e47da69a0.slice/crio-d92f514eb4fc3be26c6985b0f5ea23874782bde7f6e3e14515acaa7064ebbdc6 WatchSource:0}: Error finding container d92f514eb4fc3be26c6985b0f5ea23874782bde7f6e3e14515acaa7064ebbdc6: Status 404 returned error can't find the container with id d92f514eb4fc3be26c6985b0f5ea23874782bde7f6e3e14515acaa7064ebbdc6 Aug 04 00:50:38.776553 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.776517 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" event={"ID":"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb","Type":"ContainerStarted","Data":"329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df"} Aug 04 00:50:38.776742 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.776630 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:38.778351 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.778324 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" event={"ID":"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97","Type":"ContainerStarted","Data":"58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f"} Aug 04 00:50:38.778475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.778373 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" event={"ID":"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97","Type":"ContainerStarted","Data":"96a3882b2bf84ed3e001ddde1e4bad3e342654f4b864ac977da431b552505761"} Aug 04 00:50:38.778475 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.778402 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:38.779923 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.779901 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" event={"ID":"b091241a-3875-47f5-a6ba-958e47da69a0","Type":"ContainerStarted","Data":"13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c"} Aug 04 00:50:38.780006 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.779942 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" event={"ID":"b091241a-3875-47f5-a6ba-958e47da69a0","Type":"ContainerStarted","Data":"d92f514eb4fc3be26c6985b0f5ea23874782bde7f6e3e14515acaa7064ebbdc6"} Aug 04 00:50:38.780049 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.780021 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:50:38.796674 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.796623 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" podStartSLOduration=1.43996015 podStartE2EDuration="1.796602254s" totalImagesPullingTime="356.642104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:38.794428461 +0000 UTC m=+2876.965461769" watchObservedRunningTime="2026-08-04 00:50:38.796602254 +0000 UTC m=+2876.967635564" Aug 04 00:50:38.816055 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.816009 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" podStartSLOduration=1.8159941389999998 podStartE2EDuration="1.815994139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:38.813892873 +0000 UTC m=+2876.984926175" watchObservedRunningTime="2026-08-04 00:50:38.815994139 +0000 UTC m=+2876.987027446" Aug 04 00:50:38.832413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:38.832370 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" podStartSLOduration=1.832353041 podStartE2EDuration="1.832353041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:38.830090046 +0000 UTC m=+2877.001123354" watchObservedRunningTime="2026-08-04 00:50:38.832353041 +0000 UTC m=+2877.003386348" Aug 04 00:50:43.638698 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:43.638665 2571 scope.go:117] "RemoveContainer" containerID="1cf639771bc9c6b78879c00c7465a2c09a42f3d6c7bb88e37062527f8f405854" Aug 04 00:50:44.790250 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:44.790215 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 00:50:49.786408 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:49.786359 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" podUID="b091241a-3875-47f5-a6ba-958e47da69a0" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.82:9004: connect: connection refused" Aug 04 00:50:49.786807 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:49.786638 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 00:50:55.903264 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:55.903229 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c"] Aug 04 00:50:55.906847 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:55.906823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:55.908926 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:55.908902 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-33bda8-serving-cert\"" Aug 04 00:50:55.914545 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:55.914520 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c"] Aug 04 00:50:56.021016 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.020983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.021212 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.021083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.021212 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.021124 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqqq\" (UniqueName: \"kubernetes.io/projected/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-kube-api-access-jkqqq\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.122288 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.122248 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.122288 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.122294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqqq\" (UniqueName: \"kubernetes.io/projected/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-kube-api-access-jkqqq\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.122514 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.122336 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.122514 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:56.122456 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-serving-cert: secret "maas-api-e2e-shared-b-33bda8-serving-cert" not found Aug 04 00:50:56.122514 ip-10-0-134-15 kubenswrapper[2571]: E0804 00:50:56.122513 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls podName:24448f5b-8b6e-4e35-8813-91d0a8cdfec5 nodeName:}" failed. No retries permitted until 2026-08-04 00:50:56.622494327 +0000 UTC m=+2894.793527627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls") pod "maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" (UID: "24448f5b-8b6e-4e35-8813-91d0a8cdfec5") : secret "maas-api-e2e-shared-b-33bda8-serving-cert" not found Aug 04 00:50:56.122959 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.122935 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.132938 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.132917 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqqq\" (UniqueName: \"kubernetes.io/projected/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-kube-api-access-jkqqq\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.545000 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.544971 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j"] Aug 04 00:50:56.548272 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.548253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.550706 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.550685 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-33bda8\"" Aug 04 00:50:56.550869 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.550810 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-33bda8-dockercfg-7rjdc\"" Aug 04 00:50:56.557818 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.557792 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j"] Aug 04 00:50:56.564409 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.564388 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc"] Aug 04 00:50:56.568413 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.568393 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.576142 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.575894 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc"] Aug 04 00:50:56.627607 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.627581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.630188 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.630168 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls\") pod \"maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.728359 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.728321 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cj4\" (UniqueName: \"kubernetes.io/projected/566208e5-a439-4941-9791-24014b3dea14-kube-api-access-74cj4\") pod \"payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.728537 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.728415 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566208e5-a439-4941-9791-24014b3dea14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.728537 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.728449 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/c82a60ef-4c0c-47d2-9d91-7add0369fb73-kube-api-access-s8rgq\") pod \"payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.728537 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.728475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82a60ef-4c0c-47d2-9d91-7add0369fb73-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.819656 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.819588 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:56.828928 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.828905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cj4\" (UniqueName: \"kubernetes.io/projected/566208e5-a439-4941-9791-24014b3dea14-kube-api-access-74cj4\") pod \"payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.829033 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.828977 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566208e5-a439-4941-9791-24014b3dea14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.829033 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.829023 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/c82a60ef-4c0c-47d2-9d91-7add0369fb73-kube-api-access-s8rgq\") pod \"payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.829160 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.829057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82a60ef-4c0c-47d2-9d91-7add0369fb73-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.829801 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.829776 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82a60ef-4c0c-47d2-9d91-7add0369fb73-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.829923 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.829784 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566208e5-a439-4941-9791-24014b3dea14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.836991 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.836966 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/c82a60ef-4c0c-47d2-9d91-7add0369fb73-kube-api-access-s8rgq\") pod \"payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.837221 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.837199 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cj4\" (UniqueName: \"kubernetes.io/projected/566208e5-a439-4941-9791-24014b3dea14-kube-api-access-74cj4\") pod \"payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.859600 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.859571 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:56.879947 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.879659 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:56.979056 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:56.979024 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c"] Aug 04 00:50:57.230936 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.230905 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j"] Aug 04 00:50:57.233516 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:50:57.233476 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566208e5_a439_4941_9791_24014b3dea14.slice/crio-a79ac66f1412b09d0b5711076b25c3baf04aa4b81046e33903d16e1f372ca8a7 WatchSource:0}: Error finding container a79ac66f1412b09d0b5711076b25c3baf04aa4b81046e33903d16e1f372ca8a7: Status 404 returned error can't find the container with id a79ac66f1412b09d0b5711076b25c3baf04aa4b81046e33903d16e1f372ca8a7 Aug 04 00:50:57.261462 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.261435 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc"] Aug 04 00:50:57.263166 ip-10-0-134-15 kubenswrapper[2571]: W0804 00:50:57.263081 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82a60ef_4c0c_47d2_9d91_7add0369fb73.slice/crio-0e32dd759510d480ab9ceac353aa92af1c4dd8b88a817c64d69b4ddea8bc3d7d WatchSource:0}: Error finding container 0e32dd759510d480ab9ceac353aa92af1c4dd8b88a817c64d69b4ddea8bc3d7d: Status 404 returned error can't find the container with id 0e32dd759510d480ab9ceac353aa92af1c4dd8b88a817c64d69b4ddea8bc3d7d Aug 04 00:50:57.896374 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.896290 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" event={"ID":"24448f5b-8b6e-4e35-8813-91d0a8cdfec5","Type":"ContainerStarted","Data":"9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f"} Aug 04 00:50:57.896374 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.896353 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" event={"ID":"24448f5b-8b6e-4e35-8813-91d0a8cdfec5","Type":"ContainerStarted","Data":"9f2cc25c03cf1b03e41ad23c39b76dbca9f699f7767bb44470f1480adbcc41ce"} Aug 04 00:50:57.896596 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.896448 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:50:57.898006 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.897975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" event={"ID":"c82a60ef-4c0c-47d2-9d91-7add0369fb73","Type":"ContainerStarted","Data":"91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07"} Aug 04 00:50:57.898170 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.898033 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:50:57.898170 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.898061 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" event={"ID":"c82a60ef-4c0c-47d2-9d91-7add0369fb73","Type":"ContainerStarted","Data":"0e32dd759510d480ab9ceac353aa92af1c4dd8b88a817c64d69b4ddea8bc3d7d"} Aug 04 00:50:57.899543 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.899516 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" event={"ID":"566208e5-a439-4941-9791-24014b3dea14","Type":"ContainerStarted","Data":"1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4"} Aug 04 00:50:57.899665 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.899551 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" event={"ID":"566208e5-a439-4941-9791-24014b3dea14","Type":"ContainerStarted","Data":"a79ac66f1412b09d0b5711076b25c3baf04aa4b81046e33903d16e1f372ca8a7"} Aug 04 00:50:57.899726 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.899668 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:50:57.913289 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.913248 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" podStartSLOduration=2.351405942 podStartE2EDuration="2.913234736s" totalImagesPullingTime="561.828794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:57.91081988 +0000 UTC m=+2896.081853200" watchObservedRunningTime="2026-08-04 00:50:57.913234736 +0000 UTC m=+2896.084268042" Aug 04 00:50:57.926078 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.926036 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" podStartSLOduration=1.926023453 podStartE2EDuration="1.926023453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:57.924052835 +0000 UTC m=+2896.095086141" watchObservedRunningTime="2026-08-04 00:50:57.926023453 +0000 UTC m=+2896.097056759" Aug 04 00:50:57.940554 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:57.940510 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" podStartSLOduration=1.940493886 podStartE2EDuration="1.940493886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 00:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 00:50:57.938285757 +0000 UTC m=+2896.109319065" watchObservedRunningTime="2026-08-04 00:50:57.940493886 +0000 UTC m=+2896.111527193" Aug 04 00:50:59.785730 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:50:59.785694 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 00:51:03.910929 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:51:03.910893 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 00:51:08.907315 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:51:08.907278 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 00:51:08.907743 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:51:08.907355 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 00:54:43.712026 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:54:43.711992 2571 scope.go:117] "RemoveContainer" containerID="55168bd8f3f0658e48f9de857ed72f1ff0186424a13c20b85095e9ed3b92603a" Aug 04 00:54:43.721181 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:54:43.721164 2571 scope.go:117] "RemoveContainer" containerID="7a3fd13b090dd7a9b07b9caa36b1a986e52197dc5457a09e40634e5e886fd707" Aug 04 00:56:43.758300 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:56:43.758218 2571 scope.go:117] "RemoveContainer" containerID="8e622e2eff010f201098c0ceb79f35b41e6ecc6b06c194e71c21a080c0075978" Aug 04 00:56:43.767157 ip-10-0-134-15 kubenswrapper[2571]: I0804 00:56:43.767130 2571 scope.go:117] "RemoveContainer" containerID="945049c67ee177dfe1d3600a3dda0964f520d84db5e55b183a96555b24dcfd8d" Aug 04 01:00:00.171911 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.171817 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr"] Aug 04 01:00:00.175484 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.175454 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" Aug 04 01:00:00.178156 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.178134 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:00:00.185808 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.185784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" Aug 04 01:00:00.198181 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.198153 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr"] Aug 04 01:00:00.350554 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.350526 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr"] Aug 04 01:00:00.352454 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:00:00.352425 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44466c87_396b_4399_8d06_a7b1afccbebb.slice/crio-d05b46a84b0b0f9ae26788a7cd723bae4cb4f89a7d9af7b7ba0ef4be9a5047c4 WatchSource:0}: Error finding container d05b46a84b0b0f9ae26788a7cd723bae4cb4f89a7d9af7b7ba0ef4be9a5047c4: Status 404 returned error can't find the container with id d05b46a84b0b0f9ae26788a7cd723bae4cb4f89a7d9af7b7ba0ef4be9a5047c4 Aug 04 01:00:00.995634 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.995574 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" event={"ID":"44466c87-396b-4399-8d06-a7b1afccbebb","Type":"ContainerStarted","Data":"e62279f7d39b00fef050db4bea0cd07433b6951a09d0cc5145ed8d8b0d24f440"} Aug 04 01:00:00.995815 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:00.995666 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" event={"ID":"44466c87-396b-4399-8d06-a7b1afccbebb","Type":"ContainerStarted","Data":"d05b46a84b0b0f9ae26788a7cd723bae4cb4f89a7d9af7b7ba0ef4be9a5047c4"} Aug 04 01:00:01.011541 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:01.011493 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" podStartSLOduration=1.011476374 podStartE2EDuration="1.011476374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:01.00904936 +0000 UTC m=+3439.180082700" watchObservedRunningTime="2026-08-04 01:00:01.011476374 +0000 UTC m=+3439.182509680" Aug 04 01:00:02.002794 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:02.002759 2571 generic.go:356] "Generic (PLEG): container finished" podID="44466c87-396b-4399-8d06-a7b1afccbebb" containerID="e62279f7d39b00fef050db4bea0cd07433b6951a09d0cc5145ed8d8b0d24f440" exitCode=0 Aug 04 01:00:02.003250 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:02.002854 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" event={"ID":"44466c87-396b-4399-8d06-a7b1afccbebb","Type":"ContainerDied","Data":"e62279f7d39b00fef050db4bea0cd07433b6951a09d0cc5145ed8d8b0d24f440"} Aug 04 01:00:03.145160 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:03.145135 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" Aug 04 01:00:04.015172 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:04.015135 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" event={"ID":"44466c87-396b-4399-8d06-a7b1afccbebb","Type":"ContainerDied","Data":"d05b46a84b0b0f9ae26788a7cd723bae4cb4f89a7d9af7b7ba0ef4be9a5047c4"} Aug 04 01:00:04.015172 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:04.015155 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr" Aug 04 01:00:04.015393 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:04.015186 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05b46a84b0b0f9ae26788a7cd723bae4cb4f89a7d9af7b7ba0ef4be9a5047c4" Aug 04 01:00:28.895305 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.895271 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5"] Aug 04 01:00:28.895687 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.895676 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44466c87-396b-4399-8d06-a7b1afccbebb" containerName="cleanup" Aug 04 01:00:28.895687 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.895687 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44466c87-396b-4399-8d06-a7b1afccbebb" containerName="cleanup" Aug 04 01:00:28.895792 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.895777 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="44466c87-396b-4399-8d06-a7b1afccbebb" containerName="cleanup" Aug 04 01:00:28.899311 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.899290 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:28.901803 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.901781 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0825a6-dockercfg-5fvtg\"" Aug 04 01:00:28.901918 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.901845 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0825a6\"" Aug 04 01:00:28.910460 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.910431 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5"] Aug 04 01:00:28.962517 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.962490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpcdb\" (UniqueName: \"kubernetes.io/projected/2c1901e2-67a3-45c9-abbd-196dcf01f15a-kube-api-access-vpcdb\") pod \"payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:28.962644 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:28.962587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1901e2-67a3-45c9-abbd-196dcf01f15a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:29.063348 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:29.063309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpcdb\" (UniqueName: \"kubernetes.io/projected/2c1901e2-67a3-45c9-abbd-196dcf01f15a-kube-api-access-vpcdb\") pod \"payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:29.063535 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:29.063428 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1901e2-67a3-45c9-abbd-196dcf01f15a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:29.064380 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:29.064355 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1901e2-67a3-45c9-abbd-196dcf01f15a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:29.072240 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:29.072210 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpcdb\" (UniqueName: \"kubernetes.io/projected/2c1901e2-67a3-45c9-abbd-196dcf01f15a-kube-api-access-vpcdb\") pod \"payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:29.211497 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:29.211409 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:29.354293 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:29.354267 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5"] Aug 04 01:00:29.356435 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:00:29.356410 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1901e2_67a3_45c9_abbd_196dcf01f15a.slice/crio-340c7bdf79ab49c9bbf6e54323c714d9f6a0bbd4f9f0fa458e0e63bd602974fb WatchSource:0}: Error finding container 340c7bdf79ab49c9bbf6e54323c714d9f6a0bbd4f9f0fa458e0e63bd602974fb: Status 404 returned error can't find the container with id 340c7bdf79ab49c9bbf6e54323c714d9f6a0bbd4f9f0fa458e0e63bd602974fb Aug 04 01:00:30.169518 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:30.169479 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" event={"ID":"2c1901e2-67a3-45c9-abbd-196dcf01f15a","Type":"ContainerStarted","Data":"1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8"} Aug 04 01:00:30.169919 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:30.169540 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" event={"ID":"2c1901e2-67a3-45c9-abbd-196dcf01f15a","Type":"ContainerStarted","Data":"340c7bdf79ab49c9bbf6e54323c714d9f6a0bbd4f9f0fa458e0e63bd602974fb"} Aug 04 01:00:30.169919 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:30.169682 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:30.185491 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:30.185447 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" podStartSLOduration=2.185432192 podStartE2EDuration="2.185432192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:30.183308235 +0000 UTC m=+3468.354341544" watchObservedRunningTime="2026-08-04 01:00:30.185432192 +0000 UTC m=+3468.356465499" Aug 04 01:00:41.179149 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:41.179092 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:00:51.626931 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.626894 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2"] Aug 04 01:00:51.631722 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.631704 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.633973 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.633949 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1cbf8c\"" Aug 04 01:00:51.634159 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.634113 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1cbf8c-dockercfg-d849j\"" Aug 04 01:00:51.640603 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.640582 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2"] Aug 04 01:00:51.662090 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.662067 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf"] Aug 04 01:00:51.665534 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.665518 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.673551 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.673528 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf"] Aug 04 01:00:51.774460 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.774423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b075cb08-34a3-4073-9d17-3db73066563c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.774656 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.774483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a194fb-3a24-4324-89c9-3eb70916dee3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.774656 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.774587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2w5\" (UniqueName: \"kubernetes.io/projected/c1a194fb-3a24-4324-89c9-3eb70916dee3-kube-api-access-fp2w5\") pod \"payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.774768 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.774669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtl5\" (UniqueName: \"kubernetes.io/projected/b075cb08-34a3-4073-9d17-3db73066563c-kube-api-access-mxtl5\") pod \"payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.876049 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.876013 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a194fb-3a24-4324-89c9-3eb70916dee3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.876267 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.876068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2w5\" (UniqueName: \"kubernetes.io/projected/c1a194fb-3a24-4324-89c9-3eb70916dee3-kube-api-access-fp2w5\") pod \"payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.876267 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.876124 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtl5\" (UniqueName: \"kubernetes.io/projected/b075cb08-34a3-4073-9d17-3db73066563c-kube-api-access-mxtl5\") pod \"payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.876267 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.876162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b075cb08-34a3-4073-9d17-3db73066563c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.876834 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.876813 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b075cb08-34a3-4073-9d17-3db73066563c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.876895 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.876816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a194fb-3a24-4324-89c9-3eb70916dee3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.884092 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.884031 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtl5\" (UniqueName: \"kubernetes.io/projected/b075cb08-34a3-4073-9d17-3db73066563c-kube-api-access-mxtl5\") pod \"payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:51.884216 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.884134 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2w5\" (UniqueName: \"kubernetes.io/projected/c1a194fb-3a24-4324-89c9-3eb70916dee3-kube-api-access-fp2w5\") pod \"payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.944183 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.944132 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:51.976584 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:51.976552 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:52.105890 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.105845 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2"] Aug 04 01:00:52.112555 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:00:52.112521 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a194fb_3a24_4324_89c9_3eb70916dee3.slice/crio-89c0139d676e01ed78989f2bc1e0fd5f6b3cf39552bb489eb4168a8114ee9e90 WatchSource:0}: Error finding container 89c0139d676e01ed78989f2bc1e0fd5f6b3cf39552bb489eb4168a8114ee9e90: Status 404 returned error can't find the container with id 89c0139d676e01ed78989f2bc1e0fd5f6b3cf39552bb489eb4168a8114ee9e90 Aug 04 01:00:52.138977 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.138948 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf"] Aug 04 01:00:52.149306 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:00:52.149279 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb075cb08_34a3_4073_9d17_3db73066563c.slice/crio-eeeba6627cd0ab171dade462c7ec0b47c373351cdf453f1842010d805dd31618 WatchSource:0}: Error finding container eeeba6627cd0ab171dade462c7ec0b47c373351cdf453f1842010d805dd31618: Status 404 returned error can't find the container with id eeeba6627cd0ab171dade462c7ec0b47c373351cdf453f1842010d805dd31618 Aug 04 01:00:52.299687 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.299654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" event={"ID":"b075cb08-34a3-4073-9d17-3db73066563c","Type":"ContainerStarted","Data":"15898bde6ccc1c7ef969235e49f8f65b327c4c92b285d7cd07e5066c0121fa3b"} Aug 04 01:00:52.299878 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.299714 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" event={"ID":"b075cb08-34a3-4073-9d17-3db73066563c","Type":"ContainerStarted","Data":"eeeba6627cd0ab171dade462c7ec0b47c373351cdf453f1842010d805dd31618"} Aug 04 01:00:52.299946 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.299891 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:00:52.301484 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.301459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" event={"ID":"c1a194fb-3a24-4324-89c9-3eb70916dee3","Type":"ContainerStarted","Data":"07d04a1b5467b6fe03b96e778f162a31ecb1757e5ba7f35b499ff08c03ff7b37"} Aug 04 01:00:52.301593 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.301504 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" event={"ID":"c1a194fb-3a24-4324-89c9-3eb70916dee3","Type":"ContainerStarted","Data":"89c0139d676e01ed78989f2bc1e0fd5f6b3cf39552bb489eb4168a8114ee9e90"} Aug 04 01:00:52.301593 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.301524 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:00:52.314410 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.314363 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" podStartSLOduration=1.31434651 podStartE2EDuration="1.31434651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:52.313678523 +0000 UTC m=+3490.484711855" watchObservedRunningTime="2026-08-04 01:00:52.31434651 +0000 UTC m=+3490.485379817" Aug 04 01:00:52.328986 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:00:52.328936 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" podStartSLOduration=1.328921316 podStartE2EDuration="1.328921316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:00:52.327158061 +0000 UTC m=+3490.498191381" watchObservedRunningTime="2026-08-04 01:00:52.328921316 +0000 UTC m=+3490.499954622" Aug 04 01:01:03.308247 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:01:03.308188 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" podUID="b075cb08-34a3-4073-9d17-3db73066563c" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.89:9004: connect: connection refused" Aug 04 01:01:03.308848 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:01:03.308580 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:01:13.308189 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:01:13.308153 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:01:21.198254 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:01:21.198213 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2_c1a194fb-3a24-4324-89c9-3eb70916dee3/payload-processing/0.log" Aug 04 01:02:12.146275 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:12.146247 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2_c1a194fb-3a24-4324-89c9-3eb70916dee3/payload-processing/0.log" Aug 04 01:02:18.979434 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:18.979391 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l"] Aug 04 01:02:18.983189 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:18.983163 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:18.985427 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:18.985404 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-26288f\"" Aug 04 01:02:18.985544 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:18.985478 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-26288f-dockercfg-pjwj9\"" Aug 04 01:02:18.994460 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:18.994437 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l"] Aug 04 01:02:19.013024 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.012986 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5"] Aug 04 01:02:19.016902 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.016879 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.032179 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.031015 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5"] Aug 04 01:02:19.103317 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.103279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmfr\" (UniqueName: \"kubernetes.io/projected/68db0c1d-f59e-456c-85d2-8bfeb14adff8-kube-api-access-nzmfr\") pod \"payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.103471 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.103339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db0c1d-f59e-456c-85d2-8bfeb14adff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.204211 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.204162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db0c1d-f59e-456c-85d2-8bfeb14adff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.204428 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.204281 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9b94\" (UniqueName: \"kubernetes.io/projected/3e9717e9-e79b-4880-a2f7-a8c6da237bef-kube-api-access-p9b94\") pod \"payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.204428 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.204322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmfr\" (UniqueName: \"kubernetes.io/projected/68db0c1d-f59e-456c-85d2-8bfeb14adff8-kube-api-access-nzmfr\") pod \"payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.204428 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.204350 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9717e9-e79b-4880-a2f7-a8c6da237bef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.205052 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.205013 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db0c1d-f59e-456c-85d2-8bfeb14adff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.213361 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.213336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmfr\" (UniqueName: \"kubernetes.io/projected/68db0c1d-f59e-456c-85d2-8bfeb14adff8-kube-api-access-nzmfr\") pod \"payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.295250 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.295213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.305713 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.305684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9b94\" (UniqueName: \"kubernetes.io/projected/3e9717e9-e79b-4880-a2f7-a8c6da237bef-kube-api-access-p9b94\") pod \"payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.305859 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.305745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9717e9-e79b-4880-a2f7-a8c6da237bef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.306489 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.306470 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9717e9-e79b-4880-a2f7-a8c6da237bef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.314712 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.314689 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9b94\" (UniqueName: \"kubernetes.io/projected/3e9717e9-e79b-4880-a2f7-a8c6da237bef-kube-api-access-p9b94\") pod \"payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.329554 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.329530 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.492111 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.492073 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5"] Aug 04 01:02:19.494486 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:02:19.494452 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9717e9_e79b_4880_a2f7_a8c6da237bef.slice/crio-74a319b9204a3f74fac9df63fbca77289e30365a26fe0a1a71ffe14736f663f1 WatchSource:0}: Error finding container 74a319b9204a3f74fac9df63fbca77289e30365a26fe0a1a71ffe14736f663f1: Status 404 returned error can't find the container with id 74a319b9204a3f74fac9df63fbca77289e30365a26fe0a1a71ffe14736f663f1 Aug 04 01:02:19.654335 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.654304 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l"] Aug 04 01:02:19.656524 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:02:19.656489 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68db0c1d_f59e_456c_85d2_8bfeb14adff8.slice/crio-3b39339081a69351dc235000321feab44e1dbf0d125c4ada49d0650fbf2caaa7 WatchSource:0}: Error finding container 3b39339081a69351dc235000321feab44e1dbf0d125c4ada49d0650fbf2caaa7: Status 404 returned error can't find the container with id 3b39339081a69351dc235000321feab44e1dbf0d125c4ada49d0650fbf2caaa7 Aug 04 01:02:19.819075 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.818965 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" event={"ID":"3e9717e9-e79b-4880-a2f7-a8c6da237bef","Type":"ContainerStarted","Data":"188d9c922fbf292433943b98054447a61965c08d2a87eac9b3aeed80caa8af16"} Aug 04 01:02:19.819075 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.819043 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:19.819075 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.819070 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" event={"ID":"3e9717e9-e79b-4880-a2f7-a8c6da237bef","Type":"ContainerStarted","Data":"74a319b9204a3f74fac9df63fbca77289e30365a26fe0a1a71ffe14736f663f1"} Aug 04 01:02:19.820808 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.820777 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" event={"ID":"68db0c1d-f59e-456c-85d2-8bfeb14adff8","Type":"ContainerStarted","Data":"491ef7f4be31d1c8f71f6f2840ed9e6a1a1a91e59bfa129a328572b0f13b21c1"} Aug 04 01:02:19.820955 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.820829 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" event={"ID":"68db0c1d-f59e-456c-85d2-8bfeb14adff8","Type":"ContainerStarted","Data":"3b39339081a69351dc235000321feab44e1dbf0d125c4ada49d0650fbf2caaa7"} Aug 04 01:02:19.820955 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.820849 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:19.837450 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.837402 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" podStartSLOduration=1.837386999 podStartE2EDuration="1.837386999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:02:19.834020833 +0000 UTC m=+3578.005054173" watchObservedRunningTime="2026-08-04 01:02:19.837386999 +0000 UTC m=+3578.008420297" Aug 04 01:02:19.851972 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:19.851915 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" podStartSLOduration=1.851896233 podStartE2EDuration="1.851896233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 01:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:02:19.849907983 +0000 UTC m=+3578.020941316" watchObservedRunningTime="2026-08-04 01:02:19.851896233 +0000 UTC m=+3578.022929540" Aug 04 01:02:30.828253 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:30.828203 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" podUID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.90:9004: connect: connection refused" Aug 04 01:02:30.828848 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:30.828829 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:40.827705 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:40.827671 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:41.413170 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.413133 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4"] Aug 04 01:02:41.416909 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.416889 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:41.418923 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.418902 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:02:41.423355 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.423332 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4"] Aug 04 01:02:41.517403 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.517362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd24a6f-026c-4bb5-a2ef-3118e9b72101-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4\" (UID: \"6bd24a6f-026c-4bb5-a2ef-3118e9b72101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:41.618638 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.618597 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd24a6f-026c-4bb5-a2ef-3118e9b72101-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4\" (UID: \"6bd24a6f-026c-4bb5-a2ef-3118e9b72101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:41.619361 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.619339 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd24a6f-026c-4bb5-a2ef-3118e9b72101-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4\" (UID: \"6bd24a6f-026c-4bb5-a2ef-3118e9b72101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:41.732191 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.732084 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:41.876000 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.875968 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4"] Aug 04 01:02:41.877510 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:02:41.877484 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd24a6f_026c_4bb5_a2ef_3118e9b72101.slice/crio-a8b4aafd7570bb8cb9a0957c105b6ed635a36066f669ee53be6441ce1e8883cb WatchSource:0}: Error finding container a8b4aafd7570bb8cb9a0957c105b6ed635a36066f669ee53be6441ce1e8883cb: Status 404 returned error can't find the container with id a8b4aafd7570bb8cb9a0957c105b6ed635a36066f669ee53be6441ce1e8883cb Aug 04 01:02:41.951458 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:41.951419 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" event={"ID":"6bd24a6f-026c-4bb5-a2ef-3118e9b72101","Type":"ContainerStarted","Data":"a8b4aafd7570bb8cb9a0957c105b6ed635a36066f669ee53be6441ce1e8883cb"} Aug 04 01:02:42.959786 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:42.959695 2571 generic.go:356] "Generic (PLEG): container finished" podID="6bd24a6f-026c-4bb5-a2ef-3118e9b72101" containerID="62f8829170a7f21b4f55934480b7ef8e49ae6c2b3f73422a2d4b82ce8598cc9c" exitCode=0 Aug 04 01:02:42.960169 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:42.959776 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" event={"ID":"6bd24a6f-026c-4bb5-a2ef-3118e9b72101","Type":"ContainerDied","Data":"62f8829170a7f21b4f55934480b7ef8e49ae6c2b3f73422a2d4b82ce8598cc9c"} Aug 04 01:02:44.111652 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.111622 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:44.242333 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.242235 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd24a6f-026c-4bb5-a2ef-3118e9b72101-maas-api-service-ca\") pod \"6bd24a6f-026c-4bb5-a2ef-3118e9b72101\" (UID: \"6bd24a6f-026c-4bb5-a2ef-3118e9b72101\") " Aug 04 01:02:44.242672 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.242640 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd24a6f-026c-4bb5-a2ef-3118e9b72101-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bd24a6f-026c-4bb5-a2ef-3118e9b72101" (UID: "6bd24a6f-026c-4bb5-a2ef-3118e9b72101"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:02:44.343991 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.343955 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd24a6f-026c-4bb5-a2ef-3118e9b72101-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:02:44.974440 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.974404 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" Aug 04 01:02:44.974643 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.974391 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4" event={"ID":"6bd24a6f-026c-4bb5-a2ef-3118e9b72101","Type":"ContainerDied","Data":"a8b4aafd7570bb8cb9a0957c105b6ed635a36066f669ee53be6441ce1e8883cb"} Aug 04 01:02:44.974643 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:44.974559 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8b4aafd7570bb8cb9a0957c105b6ed635a36066f669ee53be6441ce1e8883cb" Aug 04 01:02:48.826649 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:48.826615 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5"] Aug 04 01:02:48.827146 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:48.826864 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" podUID="3e9717e9-e79b-4880-a2f7-a8c6da237bef" containerName="payload-processing" containerID="cri-o://188d9c922fbf292433943b98054447a61965c08d2a87eac9b3aeed80caa8af16" gracePeriod=30 Aug 04 01:02:48.833876 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:48.833846 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l"] Aug 04 01:02:48.834090 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:48.834065 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" podUID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" containerName="payload-pre-processing" containerID="cri-o://491ef7f4be31d1c8f71f6f2840ed9e6a1a1a91e59bfa129a328572b0f13b21c1" gracePeriod=30 Aug 04 01:02:48.980409 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:48.980365 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4"] Aug 04 01:02:48.984643 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:48.984539 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-26288f-vsrw4"] Aug 04 01:02:49.000913 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.000877 2571 generic.go:356] "Generic (PLEG): container finished" podID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" containerID="491ef7f4be31d1c8f71f6f2840ed9e6a1a1a91e59bfa129a328572b0f13b21c1" exitCode=0 Aug 04 01:02:49.001131 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.000955 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" event={"ID":"68db0c1d-f59e-456c-85d2-8bfeb14adff8","Type":"ContainerDied","Data":"491ef7f4be31d1c8f71f6f2840ed9e6a1a1a91e59bfa129a328572b0f13b21c1"} Aug 04 01:02:49.002711 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.002683 2571 generic.go:356] "Generic (PLEG): container finished" podID="3e9717e9-e79b-4880-a2f7-a8c6da237bef" containerID="188d9c922fbf292433943b98054447a61965c08d2a87eac9b3aeed80caa8af16" exitCode=0 Aug 04 01:02:49.002869 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.002717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" event={"ID":"3e9717e9-e79b-4880-a2f7-a8c6da237bef","Type":"ContainerDied","Data":"188d9c922fbf292433943b98054447a61965c08d2a87eac9b3aeed80caa8af16"} Aug 04 01:02:49.133432 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.133401 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:49.136471 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.136450 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:49.290174 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.290138 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9717e9-e79b-4880-a2f7-a8c6da237bef-plugins-config-volume\") pod \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " Aug 04 01:02:49.290174 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.290192 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzmfr\" (UniqueName: \"kubernetes.io/projected/68db0c1d-f59e-456c-85d2-8bfeb14adff8-kube-api-access-nzmfr\") pod \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " Aug 04 01:02:49.290461 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.290210 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db0c1d-f59e-456c-85d2-8bfeb14adff8-plugins-config-volume\") pod \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\" (UID: \"68db0c1d-f59e-456c-85d2-8bfeb14adff8\") " Aug 04 01:02:49.290461 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.290228 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9b94\" (UniqueName: \"kubernetes.io/projected/3e9717e9-e79b-4880-a2f7-a8c6da237bef-kube-api-access-p9b94\") pod \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\" (UID: \"3e9717e9-e79b-4880-a2f7-a8c6da237bef\") " Aug 04 01:02:49.290665 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.290636 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9717e9-e79b-4880-a2f7-a8c6da237bef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e9717e9-e79b-4880-a2f7-a8c6da237bef" (UID: "3e9717e9-e79b-4880-a2f7-a8c6da237bef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:02:49.290729 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.290641 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68db0c1d-f59e-456c-85d2-8bfeb14adff8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68db0c1d-f59e-456c-85d2-8bfeb14adff8" (UID: "68db0c1d-f59e-456c-85d2-8bfeb14adff8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:02:49.292519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.292493 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9717e9-e79b-4880-a2f7-a8c6da237bef-kube-api-access-p9b94" (OuterVolumeSpecName: "kube-api-access-p9b94") pod "3e9717e9-e79b-4880-a2f7-a8c6da237bef" (UID: "3e9717e9-e79b-4880-a2f7-a8c6da237bef"). InnerVolumeSpecName "kube-api-access-p9b94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:02:49.292653 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.292523 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db0c1d-f59e-456c-85d2-8bfeb14adff8-kube-api-access-nzmfr" (OuterVolumeSpecName: "kube-api-access-nzmfr") pod "68db0c1d-f59e-456c-85d2-8bfeb14adff8" (UID: "68db0c1d-f59e-456c-85d2-8bfeb14adff8"). InnerVolumeSpecName "kube-api-access-nzmfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:02:49.391690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.391589 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9717e9-e79b-4880-a2f7-a8c6da237bef-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:02:49.391690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.391619 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzmfr\" (UniqueName: \"kubernetes.io/projected/68db0c1d-f59e-456c-85d2-8bfeb14adff8-kube-api-access-nzmfr\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:02:49.391690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.391629 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68db0c1d-f59e-456c-85d2-8bfeb14adff8-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:02:49.391690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:49.391639 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9b94\" (UniqueName: \"kubernetes.io/projected/3e9717e9-e79b-4880-a2f7-a8c6da237bef-kube-api-access-p9b94\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:02:50.011355 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.011310 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" event={"ID":"3e9717e9-e79b-4880-a2f7-a8c6da237bef","Type":"ContainerDied","Data":"74a319b9204a3f74fac9df63fbca77289e30365a26fe0a1a71ffe14736f663f1"} Aug 04 01:02:50.011869 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.011323 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5" Aug 04 01:02:50.011869 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.011388 2571 scope.go:117] "RemoveContainer" containerID="188d9c922fbf292433943b98054447a61965c08d2a87eac9b3aeed80caa8af16" Aug 04 01:02:50.019118 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.013511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" event={"ID":"68db0c1d-f59e-456c-85d2-8bfeb14adff8","Type":"ContainerDied","Data":"3b39339081a69351dc235000321feab44e1dbf0d125c4ada49d0650fbf2caaa7"} Aug 04 01:02:50.019118 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.013666 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l" Aug 04 01:02:50.032130 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.032091 2571 scope.go:117] "RemoveContainer" containerID="491ef7f4be31d1c8f71f6f2840ed9e6a1a1a91e59bfa129a328572b0f13b21c1" Aug 04 01:02:50.040968 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.040925 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5"] Aug 04 01:02:50.043847 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.043823 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-26288f-7c5d997c68-s7pg5"] Aug 04 01:02:50.052741 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.052719 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l"] Aug 04 01:02:50.055178 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.055156 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-26288f-6497d8fdf6-rl86l"] Aug 04 01:02:50.377896 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.377861 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9717e9-e79b-4880-a2f7-a8c6da237bef" path="/var/lib/kubelet/pods/3e9717e9-e79b-4880-a2f7-a8c6da237bef/volumes" Aug 04 01:02:50.378270 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.378256 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" path="/var/lib/kubelet/pods/68db0c1d-f59e-456c-85d2-8bfeb14adff8/volumes" Aug 04 01:02:50.378554 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:50.378540 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd24a6f-026c-4bb5-a2ef-3118e9b72101" path="/var/lib/kubelet/pods/6bd24a6f-026c-4bb5-a2ef-3118e9b72101/volumes" Aug 04 01:02:59.695483 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695388 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n"] Aug 04 01:02:59.695920 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695905 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9717e9-e79b-4880-a2f7-a8c6da237bef" containerName="payload-processing" Aug 04 01:02:59.695975 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695922 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9717e9-e79b-4880-a2f7-a8c6da237bef" containerName="payload-processing" Aug 04 01:02:59.695975 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695934 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" containerName="payload-pre-processing" Aug 04 01:02:59.695975 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695940 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" containerName="payload-pre-processing" Aug 04 01:02:59.695975 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695960 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd24a6f-026c-4bb5-a2ef-3118e9b72101" containerName="revoke-keys" Aug 04 01:02:59.695975 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.695967 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd24a6f-026c-4bb5-a2ef-3118e9b72101" containerName="revoke-keys" Aug 04 01:02:59.696169 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.696053 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd24a6f-026c-4bb5-a2ef-3118e9b72101" containerName="revoke-keys" Aug 04 01:02:59.696169 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.696061 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="68db0c1d-f59e-456c-85d2-8bfeb14adff8" containerName="payload-pre-processing" Aug 04 01:02:59.696169 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.696067 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9717e9-e79b-4880-a2f7-a8c6da237bef" containerName="payload-processing" Aug 04 01:02:59.699138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.699097 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:02:59.702194 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.702170 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:02:59.708596 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.707907 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n"] Aug 04 01:02:59.784266 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.784214 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931af648-ae00-450d-8970-faf034966458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n\" (UID: \"931af648-ae00-450d-8970-faf034966458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:02:59.885232 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.885191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931af648-ae00-450d-8970-faf034966458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n\" (UID: \"931af648-ae00-450d-8970-faf034966458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:02:59.885896 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:02:59.885872 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931af648-ae00-450d-8970-faf034966458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n\" (UID: \"931af648-ae00-450d-8970-faf034966458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:03:00.017288 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:00.017188 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:03:00.158634 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:00.158595 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n"] Aug 04 01:03:00.160088 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:03:00.160061 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931af648_ae00_450d_8970_faf034966458.slice/crio-ca987c09b7a5297a958aadf90ab580ac87e221a5dba1a1b0288de67448e1eb2a WatchSource:0}: Error finding container ca987c09b7a5297a958aadf90ab580ac87e221a5dba1a1b0288de67448e1eb2a: Status 404 returned error can't find the container with id ca987c09b7a5297a958aadf90ab580ac87e221a5dba1a1b0288de67448e1eb2a Aug 04 01:03:01.085071 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:01.085031 2571 generic.go:356] "Generic (PLEG): container finished" podID="931af648-ae00-450d-8970-faf034966458" containerID="74b1b40b6a747e0ec63c3a0358530c12d5fc92179b811ebb187b0a7b85f6fed0" exitCode=0 Aug 04 01:03:01.085499 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:01.085087 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" event={"ID":"931af648-ae00-450d-8970-faf034966458","Type":"ContainerDied","Data":"74b1b40b6a747e0ec63c3a0358530c12d5fc92179b811ebb187b0a7b85f6fed0"} Aug 04 01:03:01.085499 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:01.085153 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" event={"ID":"931af648-ae00-450d-8970-faf034966458","Type":"ContainerStarted","Data":"ca987c09b7a5297a958aadf90ab580ac87e221a5dba1a1b0288de67448e1eb2a"} Aug 04 01:03:02.226833 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:02.226807 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:03:02.307571 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:02.307534 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931af648-ae00-450d-8970-faf034966458-maas-api-service-ca\") pod \"931af648-ae00-450d-8970-faf034966458\" (UID: \"931af648-ae00-450d-8970-faf034966458\") " Aug 04 01:03:02.307946 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:02.307919 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931af648-ae00-450d-8970-faf034966458-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "931af648-ae00-450d-8970-faf034966458" (UID: "931af648-ae00-450d-8970-faf034966458"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:03:02.408337 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:02.408293 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/931af648-ae00-450d-8970-faf034966458-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:03:03.098898 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:03.098863 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" Aug 04 01:03:03.099079 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:03.098894 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n" event={"ID":"931af648-ae00-450d-8970-faf034966458","Type":"ContainerDied","Data":"ca987c09b7a5297a958aadf90ab580ac87e221a5dba1a1b0288de67448e1eb2a"} Aug 04 01:03:03.099079 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:03.098944 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca987c09b7a5297a958aadf90ab580ac87e221a5dba1a1b0288de67448e1eb2a" Aug 04 01:03:16.608592 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.608553 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5"] Aug 04 01:03:16.609134 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.608789 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" podUID="2c1901e2-67a3-45c9-abbd-196dcf01f15a" containerName="payload-pre-processing" containerID="cri-o://1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8" gracePeriod=30 Aug 04 01:03:16.767182 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.767146 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n"] Aug 04 01:03:16.769912 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.769878 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0825a6-wmv7n"] Aug 04 01:03:16.885082 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.885055 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:03:16.933814 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.933783 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpcdb\" (UniqueName: \"kubernetes.io/projected/2c1901e2-67a3-45c9-abbd-196dcf01f15a-kube-api-access-vpcdb\") pod \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " Aug 04 01:03:16.934010 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.933829 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1901e2-67a3-45c9-abbd-196dcf01f15a-plugins-config-volume\") pod \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\" (UID: \"2c1901e2-67a3-45c9-abbd-196dcf01f15a\") " Aug 04 01:03:16.934251 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.934226 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1901e2-67a3-45c9-abbd-196dcf01f15a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c1901e2-67a3-45c9-abbd-196dcf01f15a" (UID: "2c1901e2-67a3-45c9-abbd-196dcf01f15a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:03:16.935883 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:16.935861 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1901e2-67a3-45c9-abbd-196dcf01f15a-kube-api-access-vpcdb" (OuterVolumeSpecName: "kube-api-access-vpcdb") pod "2c1901e2-67a3-45c9-abbd-196dcf01f15a" (UID: "2c1901e2-67a3-45c9-abbd-196dcf01f15a"). InnerVolumeSpecName "kube-api-access-vpcdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:03:17.034468 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.034429 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpcdb\" (UniqueName: \"kubernetes.io/projected/2c1901e2-67a3-45c9-abbd-196dcf01f15a-kube-api-access-vpcdb\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.034468 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.034460 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1901e2-67a3-45c9-abbd-196dcf01f15a-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:03:17.198484 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.198387 2571 generic.go:356] "Generic (PLEG): container finished" podID="2c1901e2-67a3-45c9-abbd-196dcf01f15a" containerID="1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8" exitCode=0 Aug 04 01:03:17.198484 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.198446 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" Aug 04 01:03:17.198716 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.198476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" event={"ID":"2c1901e2-67a3-45c9-abbd-196dcf01f15a","Type":"ContainerDied","Data":"1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8"} Aug 04 01:03:17.198716 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.198539 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5" event={"ID":"2c1901e2-67a3-45c9-abbd-196dcf01f15a","Type":"ContainerDied","Data":"340c7bdf79ab49c9bbf6e54323c714d9f6a0bbd4f9f0fa458e0e63bd602974fb"} Aug 04 01:03:17.198716 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.198575 2571 scope.go:117] "RemoveContainer" containerID="1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8" Aug 04 01:03:17.208577 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.208550 2571 scope.go:117] "RemoveContainer" containerID="1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8" Aug 04 01:03:17.208849 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:03:17.208826 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8\": container with ID starting with 1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8 not found: ID does not exist" containerID="1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8" Aug 04 01:03:17.208929 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.208860 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8"} err="failed to get container status \"1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8\": rpc error: code = NotFound desc = could not find container \"1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8\": container with ID starting with 1c68ee5e449a675767a0c9aa5ad404542f240d15fb465b4d94a5e797cb5f0ea8 not found: ID does not exist" Aug 04 01:03:17.219760 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.219733 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5"] Aug 04 01:03:17.223779 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:17.223751 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0825a6-85bb98c764-rc4v5"] Aug 04 01:03:18.377995 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:18.377955 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1901e2-67a3-45c9-abbd-196dcf01f15a" path="/var/lib/kubelet/pods/2c1901e2-67a3-45c9-abbd-196dcf01f15a/volumes" Aug 04 01:03:18.378455 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:03:18.378358 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931af648-ae00-450d-8970-faf034966458" path="/var/lib/kubelet/pods/931af648-ae00-450d-8970-faf034966458/volumes" Aug 04 01:04:21.095998 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.095961 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx"] Aug 04 01:04:21.096477 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.096441 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931af648-ae00-450d-8970-faf034966458" containerName="revoke-keys" Aug 04 01:04:21.096477 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.096458 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="931af648-ae00-450d-8970-faf034966458" containerName="revoke-keys" Aug 04 01:04:21.096477 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.096468 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1901e2-67a3-45c9-abbd-196dcf01f15a" containerName="payload-pre-processing" Aug 04 01:04:21.096477 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.096474 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1901e2-67a3-45c9-abbd-196dcf01f15a" containerName="payload-pre-processing" Aug 04 01:04:21.096624 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.096563 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="931af648-ae00-450d-8970-faf034966458" containerName="revoke-keys" Aug 04 01:04:21.096624 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.096574 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1901e2-67a3-45c9-abbd-196dcf01f15a" containerName="payload-pre-processing" Aug 04 01:04:21.099924 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.099905 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:21.102291 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.102270 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:04:21.106404 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.106384 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx"] Aug 04 01:04:21.126735 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.126707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba4cd9-b2d6-4ea1-8565-811068d22321-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx\" (UID: \"26ba4cd9-b2d6-4ea1-8565-811068d22321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:21.227448 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.227409 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba4cd9-b2d6-4ea1-8565-811068d22321-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx\" (UID: \"26ba4cd9-b2d6-4ea1-8565-811068d22321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:21.228183 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.228156 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba4cd9-b2d6-4ea1-8565-811068d22321-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx\" (UID: \"26ba4cd9-b2d6-4ea1-8565-811068d22321\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:21.411982 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.411900 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:21.549320 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.549292 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx"] Aug 04 01:04:21.551626 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:04:21.551596 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ba4cd9_b2d6_4ea1_8565_811068d22321.slice/crio-2343dd5b00053edc1ba53e8fe2d16b530747ebda870235cca27ad1c60d790e9a WatchSource:0}: Error finding container 2343dd5b00053edc1ba53e8fe2d16b530747ebda870235cca27ad1c60d790e9a: Status 404 returned error can't find the container with id 2343dd5b00053edc1ba53e8fe2d16b530747ebda870235cca27ad1c60d790e9a Aug 04 01:04:21.567289 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:21.567263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" event={"ID":"26ba4cd9-b2d6-4ea1-8565-811068d22321","Type":"ContainerStarted","Data":"2343dd5b00053edc1ba53e8fe2d16b530747ebda870235cca27ad1c60d790e9a"} Aug 04 01:04:22.574754 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:22.574724 2571 generic.go:356] "Generic (PLEG): container finished" podID="26ba4cd9-b2d6-4ea1-8565-811068d22321" containerID="cba081123bfeab9c80376c1acbbc46cb01ff086dba181043bdca5a43c2a2606e" exitCode=0 Aug 04 01:04:22.575157 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:22.574776 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" event={"ID":"26ba4cd9-b2d6-4ea1-8565-811068d22321","Type":"ContainerDied","Data":"cba081123bfeab9c80376c1acbbc46cb01ff086dba181043bdca5a43c2a2606e"} Aug 04 01:04:23.720229 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:23.720196 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:23.749189 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:23.749150 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba4cd9-b2d6-4ea1-8565-811068d22321-maas-api-service-ca\") pod \"26ba4cd9-b2d6-4ea1-8565-811068d22321\" (UID: \"26ba4cd9-b2d6-4ea1-8565-811068d22321\") " Aug 04 01:04:23.749706 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:23.749674 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ba4cd9-b2d6-4ea1-8565-811068d22321-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26ba4cd9-b2d6-4ea1-8565-811068d22321" (UID: "26ba4cd9-b2d6-4ea1-8565-811068d22321"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:23.850857 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:23.850769 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ba4cd9-b2d6-4ea1-8565-811068d22321-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:24.587118 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:24.587082 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" Aug 04 01:04:24.587293 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:24.587080 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx" event={"ID":"26ba4cd9-b2d6-4ea1-8565-811068d22321","Type":"ContainerDied","Data":"2343dd5b00053edc1ba53e8fe2d16b530747ebda870235cca27ad1c60d790e9a"} Aug 04 01:04:24.587293 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:24.587224 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2343dd5b00053edc1ba53e8fe2d16b530747ebda870235cca27ad1c60d790e9a" Aug 04 01:04:28.429906 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.429862 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2"] Aug 04 01:04:28.430431 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.430134 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" podUID="c1a194fb-3a24-4324-89c9-3eb70916dee3" containerName="payload-processing" containerID="cri-o://07d04a1b5467b6fe03b96e778f162a31ecb1757e5ba7f35b499ff08c03ff7b37" gracePeriod=30 Aug 04 01:04:28.433706 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.433670 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf"] Aug 04 01:04:28.434085 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.434053 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" podUID="b075cb08-34a3-4073-9d17-3db73066563c" containerName="payload-pre-processing" containerID="cri-o://15898bde6ccc1c7ef969235e49f8f65b327c4c92b285d7cd07e5066c0121fa3b" gracePeriod=30 Aug 04 01:04:28.577963 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.577927 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx"] Aug 04 01:04:28.581966 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.581936 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1cbf8c-t9jlx"] Aug 04 01:04:28.619729 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.617723 2571 generic.go:356] "Generic (PLEG): container finished" podID="b075cb08-34a3-4073-9d17-3db73066563c" containerID="15898bde6ccc1c7ef969235e49f8f65b327c4c92b285d7cd07e5066c0121fa3b" exitCode=0 Aug 04 01:04:28.619729 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.617918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" event={"ID":"b075cb08-34a3-4073-9d17-3db73066563c","Type":"ContainerDied","Data":"15898bde6ccc1c7ef969235e49f8f65b327c4c92b285d7cd07e5066c0121fa3b"} Aug 04 01:04:28.619951 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.619820 2571 generic.go:356] "Generic (PLEG): container finished" podID="c1a194fb-3a24-4324-89c9-3eb70916dee3" containerID="07d04a1b5467b6fe03b96e778f162a31ecb1757e5ba7f35b499ff08c03ff7b37" exitCode=0 Aug 04 01:04:28.619951 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.619927 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" event={"ID":"c1a194fb-3a24-4324-89c9-3eb70916dee3","Type":"ContainerDied","Data":"07d04a1b5467b6fe03b96e778f162a31ecb1757e5ba7f35b499ff08c03ff7b37"} Aug 04 01:04:28.695567 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.695494 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:04:28.730565 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.730537 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:04:28.797077 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.797046 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp2w5\" (UniqueName: \"kubernetes.io/projected/c1a194fb-3a24-4324-89c9-3eb70916dee3-kube-api-access-fp2w5\") pod \"c1a194fb-3a24-4324-89c9-3eb70916dee3\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " Aug 04 01:04:28.797077 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.797092 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a194fb-3a24-4324-89c9-3eb70916dee3-plugins-config-volume\") pod \"c1a194fb-3a24-4324-89c9-3eb70916dee3\" (UID: \"c1a194fb-3a24-4324-89c9-3eb70916dee3\") " Aug 04 01:04:28.797325 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.797158 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxtl5\" (UniqueName: \"kubernetes.io/projected/b075cb08-34a3-4073-9d17-3db73066563c-kube-api-access-mxtl5\") pod \"b075cb08-34a3-4073-9d17-3db73066563c\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " Aug 04 01:04:28.797325 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.797187 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b075cb08-34a3-4073-9d17-3db73066563c-plugins-config-volume\") pod \"b075cb08-34a3-4073-9d17-3db73066563c\" (UID: \"b075cb08-34a3-4073-9d17-3db73066563c\") " Aug 04 01:04:28.797588 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.797554 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a194fb-3a24-4324-89c9-3eb70916dee3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1a194fb-3a24-4324-89c9-3eb70916dee3" (UID: "c1a194fb-3a24-4324-89c9-3eb70916dee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:28.797708 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.797601 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b075cb08-34a3-4073-9d17-3db73066563c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b075cb08-34a3-4073-9d17-3db73066563c" (UID: "b075cb08-34a3-4073-9d17-3db73066563c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:28.799268 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.799246 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b075cb08-34a3-4073-9d17-3db73066563c-kube-api-access-mxtl5" (OuterVolumeSpecName: "kube-api-access-mxtl5") pod "b075cb08-34a3-4073-9d17-3db73066563c" (UID: "b075cb08-34a3-4073-9d17-3db73066563c"). InnerVolumeSpecName "kube-api-access-mxtl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:28.799332 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.799254 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a194fb-3a24-4324-89c9-3eb70916dee3-kube-api-access-fp2w5" (OuterVolumeSpecName: "kube-api-access-fp2w5") pod "c1a194fb-3a24-4324-89c9-3eb70916dee3" (UID: "c1a194fb-3a24-4324-89c9-3eb70916dee3"). InnerVolumeSpecName "kube-api-access-fp2w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:28.897974 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.897924 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2w5\" (UniqueName: \"kubernetes.io/projected/c1a194fb-3a24-4324-89c9-3eb70916dee3-kube-api-access-fp2w5\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:28.897974 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.897968 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a194fb-3a24-4324-89c9-3eb70916dee3-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:28.897974 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.897980 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtl5\" (UniqueName: \"kubernetes.io/projected/b075cb08-34a3-4073-9d17-3db73066563c-kube-api-access-mxtl5\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:28.897974 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:28.897989 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b075cb08-34a3-4073-9d17-3db73066563c-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:29.626943 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.626909 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" Aug 04 01:04:29.626943 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.626933 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2" event={"ID":"c1a194fb-3a24-4324-89c9-3eb70916dee3","Type":"ContainerDied","Data":"89c0139d676e01ed78989f2bc1e0fd5f6b3cf39552bb489eb4168a8114ee9e90"} Aug 04 01:04:29.627527 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.627015 2571 scope.go:117] "RemoveContainer" containerID="07d04a1b5467b6fe03b96e778f162a31ecb1757e5ba7f35b499ff08c03ff7b37" Aug 04 01:04:29.628477 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.628386 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" Aug 04 01:04:29.628477 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.628401 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf" event={"ID":"b075cb08-34a3-4073-9d17-3db73066563c","Type":"ContainerDied","Data":"eeeba6627cd0ab171dade462c7ec0b47c373351cdf453f1842010d805dd31618"} Aug 04 01:04:29.637993 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.637969 2571 scope.go:117] "RemoveContainer" containerID="15898bde6ccc1c7ef969235e49f8f65b327c4c92b285d7cd07e5066c0121fa3b" Aug 04 01:04:29.653489 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.653461 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf"] Aug 04 01:04:29.657331 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.657298 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1cbf8c-55585cb676-v2dlf"] Aug 04 01:04:29.668131 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.668091 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2"] Aug 04 01:04:29.670959 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:29.670938 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1cbf8c-ffb84fc54-krnl2"] Aug 04 01:04:30.377695 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:30.377661 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ba4cd9-b2d6-4ea1-8565-811068d22321" path="/var/lib/kubelet/pods/26ba4cd9-b2d6-4ea1-8565-811068d22321/volumes" Aug 04 01:04:30.378168 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:30.378144 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b075cb08-34a3-4073-9d17-3db73066563c" path="/var/lib/kubelet/pods/b075cb08-34a3-4073-9d17-3db73066563c/volumes" Aug 04 01:04:30.378574 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:30.378558 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a194fb-3a24-4324-89c9-3eb70916dee3" path="/var/lib/kubelet/pods/c1a194fb-3a24-4324-89c9-3eb70916dee3/volumes" Aug 04 01:04:35.443290 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443254 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs"] Aug 04 01:04:35.443690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443644 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ba4cd9-b2d6-4ea1-8565-811068d22321" containerName="revoke-keys" Aug 04 01:04:35.443690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443655 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ba4cd9-b2d6-4ea1-8565-811068d22321" containerName="revoke-keys" Aug 04 01:04:35.443690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443672 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a194fb-3a24-4324-89c9-3eb70916dee3" containerName="payload-processing" Aug 04 01:04:35.443690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443679 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a194fb-3a24-4324-89c9-3eb70916dee3" containerName="payload-processing" Aug 04 01:04:35.443690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443687 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b075cb08-34a3-4073-9d17-3db73066563c" containerName="payload-pre-processing" Aug 04 01:04:35.443690 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443693 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b075cb08-34a3-4073-9d17-3db73066563c" containerName="payload-pre-processing" Aug 04 01:04:35.443931 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443760 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b075cb08-34a3-4073-9d17-3db73066563c" containerName="payload-pre-processing" Aug 04 01:04:35.443931 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443770 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ba4cd9-b2d6-4ea1-8565-811068d22321" containerName="revoke-keys" Aug 04 01:04:35.443931 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.443778 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a194fb-3a24-4324-89c9-3eb70916dee3" containerName="payload-processing" Aug 04 01:04:35.446929 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.446913 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:35.449877 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.449860 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:04:35.455755 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.455720 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs"] Aug 04 01:04:35.562035 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.561999 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c03320-ffc2-4f6c-b128-872e4c506776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs\" (UID: \"89c03320-ffc2-4f6c-b128-872e4c506776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:35.663069 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.663034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c03320-ffc2-4f6c-b128-872e4c506776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs\" (UID: \"89c03320-ffc2-4f6c-b128-872e4c506776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:35.663736 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.663713 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c03320-ffc2-4f6c-b128-872e4c506776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs\" (UID: \"89c03320-ffc2-4f6c-b128-872e4c506776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:35.759653 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.759554 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:35.900573 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:35.900540 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs"] Aug 04 01:04:35.901933 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:04:35.901908 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c03320_ffc2_4f6c_b128_872e4c506776.slice/crio-c052c56d769caea184c6a44d265c59201a22af94b7c381646a3691d891678f28 WatchSource:0}: Error finding container c052c56d769caea184c6a44d265c59201a22af94b7c381646a3691d891678f28: Status 404 returned error can't find the container with id c052c56d769caea184c6a44d265c59201a22af94b7c381646a3691d891678f28 Aug 04 01:04:36.675719 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:36.675676 2571 generic.go:356] "Generic (PLEG): container finished" podID="89c03320-ffc2-4f6c-b128-872e4c506776" containerID="2d2f156b18077d382ca536c4e02e7c77402221232ccfe5663baaa2d95fe4ec9f" exitCode=0 Aug 04 01:04:36.676091 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:36.675732 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" event={"ID":"89c03320-ffc2-4f6c-b128-872e4c506776","Type":"ContainerDied","Data":"2d2f156b18077d382ca536c4e02e7c77402221232ccfe5663baaa2d95fe4ec9f"} Aug 04 01:04:36.676091 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:36.675781 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" event={"ID":"89c03320-ffc2-4f6c-b128-872e4c506776","Type":"ContainerStarted","Data":"c052c56d769caea184c6a44d265c59201a22af94b7c381646a3691d891678f28"} Aug 04 01:04:37.833782 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:37.833756 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:37.984550 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:37.984454 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c03320-ffc2-4f6c-b128-872e4c506776-maas-api-service-ca\") pod \"89c03320-ffc2-4f6c-b128-872e4c506776\" (UID: \"89c03320-ffc2-4f6c-b128-872e4c506776\") " Aug 04 01:04:37.984838 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:37.984812 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c03320-ffc2-4f6c-b128-872e4c506776-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89c03320-ffc2-4f6c-b128-872e4c506776" (UID: "89c03320-ffc2-4f6c-b128-872e4c506776"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:38.085941 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:38.085900 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c03320-ffc2-4f6c-b128-872e4c506776-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:38.688016 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:38.687915 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" Aug 04 01:04:38.688016 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:38.687976 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs" event={"ID":"89c03320-ffc2-4f6c-b128-872e4c506776","Type":"ContainerDied","Data":"c052c56d769caea184c6a44d265c59201a22af94b7c381646a3691d891678f28"} Aug 04 01:04:38.688316 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:38.688036 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c052c56d769caea184c6a44d265c59201a22af94b7c381646a3691d891678f28" Aug 04 01:04:41.075388 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.075349 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz"] Aug 04 01:04:41.075905 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.075644 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" podUID="a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" containerName="maas-api" containerID="cri-o://329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df" gracePeriod=30 Aug 04 01:04:41.110428 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.110384 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr"] Aug 04 01:04:41.113075 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.113045 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ddf6e-29763420-v5mdr"] Aug 04 01:04:41.140865 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.140831 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96"] Aug 04 01:04:41.141149 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.141079 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" podUID="72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" containerName="payload-processing" containerID="cri-o://58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f" gracePeriod=30 Aug 04 01:04:41.146954 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.146921 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt"] Aug 04 01:04:41.147205 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.147176 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" podUID="b091241a-3875-47f5-a6ba-958e47da69a0" containerName="payload-pre-processing" containerID="cri-o://13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c" gracePeriod=30 Aug 04 01:04:41.279259 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.279221 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs"] Aug 04 01:04:41.281938 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.281907 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5ddf6e-2kjvs"] Aug 04 01:04:41.413078 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.413050 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 01:04:41.459300 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.459276 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 01:04:41.462460 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.462442 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 01:04:41.517901 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.517872 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t72lh\" (UniqueName: \"kubernetes.io/projected/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-kube-api-access-t72lh\") pod \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " Aug 04 01:04:41.518068 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.517907 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-maas-api-tls\") pod \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " Aug 04 01:04:41.518068 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.517999 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-openshift-service-ca\") pod \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\" (UID: \"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb\") " Aug 04 01:04:41.518414 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.518385 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" (UID: "a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:41.520163 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.520135 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" (UID: "a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:04:41.520263 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.520195 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-kube-api-access-t72lh" (OuterVolumeSpecName: "kube-api-access-t72lh") pod "a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" (UID: "a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb"). InnerVolumeSpecName "kube-api-access-t72lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:41.618983 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.618943 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fzcc\" (UniqueName: \"kubernetes.io/projected/b091241a-3875-47f5-a6ba-958e47da69a0-kube-api-access-9fzcc\") pod \"b091241a-3875-47f5-a6ba-958e47da69a0\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " Aug 04 01:04:41.619203 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619017 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42nj5\" (UniqueName: \"kubernetes.io/projected/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-kube-api-access-42nj5\") pod \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " Aug 04 01:04:41.619203 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619068 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-plugins-config-volume\") pod \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\" (UID: \"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97\") " Aug 04 01:04:41.619203 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619149 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b091241a-3875-47f5-a6ba-958e47da69a0-plugins-config-volume\") pod \"b091241a-3875-47f5-a6ba-958e47da69a0\" (UID: \"b091241a-3875-47f5-a6ba-958e47da69a0\") " Aug 04 01:04:41.619434 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619415 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72lh\" (UniqueName: \"kubernetes.io/projected/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-kube-api-access-t72lh\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.619492 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619440 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.619492 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619455 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.619564 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619475 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" (UID: "72fe2a82-f6ba-4c6b-b5bb-624fa5337d97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:41.619564 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.619514 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b091241a-3875-47f5-a6ba-958e47da69a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b091241a-3875-47f5-a6ba-958e47da69a0" (UID: "b091241a-3875-47f5-a6ba-958e47da69a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:41.621187 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.621165 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-kube-api-access-42nj5" (OuterVolumeSpecName: "kube-api-access-42nj5") pod "72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" (UID: "72fe2a82-f6ba-4c6b-b5bb-624fa5337d97"). InnerVolumeSpecName "kube-api-access-42nj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:41.621187 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.621183 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091241a-3875-47f5-a6ba-958e47da69a0-kube-api-access-9fzcc" (OuterVolumeSpecName: "kube-api-access-9fzcc") pod "b091241a-3875-47f5-a6ba-958e47da69a0" (UID: "b091241a-3875-47f5-a6ba-958e47da69a0"). InnerVolumeSpecName "kube-api-access-9fzcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:41.707717 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.707633 2571 generic.go:356] "Generic (PLEG): container finished" podID="b091241a-3875-47f5-a6ba-958e47da69a0" containerID="13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c" exitCode=0 Aug 04 01:04:41.707717 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.707687 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" Aug 04 01:04:41.707926 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.707720 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" event={"ID":"b091241a-3875-47f5-a6ba-958e47da69a0","Type":"ContainerDied","Data":"13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c"} Aug 04 01:04:41.707926 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.707778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt" event={"ID":"b091241a-3875-47f5-a6ba-958e47da69a0","Type":"ContainerDied","Data":"d92f514eb4fc3be26c6985b0f5ea23874782bde7f6e3e14515acaa7064ebbdc6"} Aug 04 01:04:41.707926 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.707809 2571 scope.go:117] "RemoveContainer" containerID="13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c" Aug 04 01:04:41.709211 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.709178 2571 generic.go:356] "Generic (PLEG): container finished" podID="a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" containerID="329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df" exitCode=0 Aug 04 01:04:41.709339 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.709237 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" Aug 04 01:04:41.709339 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.709263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" event={"ID":"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb","Type":"ContainerDied","Data":"329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df"} Aug 04 01:04:41.709339 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.709316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz" event={"ID":"a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb","Type":"ContainerDied","Data":"b1893f1eb93d229dcd02049ef9efc25f38ab97d318cf0bbba45155066ef7427d"} Aug 04 01:04:41.710824 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.710802 2571 generic.go:356] "Generic (PLEG): container finished" podID="72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" containerID="58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f" exitCode=0 Aug 04 01:04:41.710949 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.710877 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" event={"ID":"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97","Type":"ContainerDied","Data":"58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f"} Aug 04 01:04:41.710949 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.710918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" event={"ID":"72fe2a82-f6ba-4c6b-b5bb-624fa5337d97","Type":"ContainerDied","Data":"96a3882b2bf84ed3e001ddde1e4bad3e342654f4b864ac977da431b552505761"} Aug 04 01:04:41.710949 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.710925 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96" Aug 04 01:04:41.719516 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719483 2571 scope.go:117] "RemoveContainer" containerID="13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c" Aug 04 01:04:41.719981 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:04:41.719849 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c\": container with ID starting with 13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c not found: ID does not exist" containerID="13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c" Aug 04 01:04:41.719981 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719877 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c"} err="failed to get container status \"13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c\": rpc error: code = NotFound desc = could not find container \"13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c\": container with ID starting with 13ea49cb07ab0c15c20e8cee941bdd6f5662b76f368f2aa996fa655c4f52402c not found: ID does not exist" Aug 04 01:04:41.719981 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719894 2571 scope.go:117] "RemoveContainer" containerID="329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df" Aug 04 01:04:41.719981 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719935 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzcc\" (UniqueName: \"kubernetes.io/projected/b091241a-3875-47f5-a6ba-958e47da69a0-kube-api-access-9fzcc\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.719981 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719958 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42nj5\" (UniqueName: \"kubernetes.io/projected/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-kube-api-access-42nj5\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.719981 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719975 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.720356 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.719990 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b091241a-3875-47f5-a6ba-958e47da69a0-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:41.729375 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.729359 2571 scope.go:117] "RemoveContainer" containerID="329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df" Aug 04 01:04:41.729643 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:04:41.729622 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df\": container with ID starting with 329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df not found: ID does not exist" containerID="329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df" Aug 04 01:04:41.729707 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.729649 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df"} err="failed to get container status \"329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df\": rpc error: code = NotFound desc = could not find container \"329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df\": container with ID starting with 329451fcd0525ed95d0514c09504e7d2ef6f5e0b77bb0be1e4fe7b71f104d6df not found: ID does not exist" Aug 04 01:04:41.729707 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.729666 2571 scope.go:117] "RemoveContainer" containerID="58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f" Aug 04 01:04:41.736341 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.736320 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt"] Aug 04 01:04:41.739432 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.739414 2571 scope.go:117] "RemoveContainer" containerID="58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f" Aug 04 01:04:41.739693 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:04:41.739675 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f\": container with ID starting with 58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f not found: ID does not exist" containerID="58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f" Aug 04 01:04:41.739780 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.739700 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f"} err="failed to get container status \"58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f\": rpc error: code = NotFound desc = could not find container \"58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f\": container with ID starting with 58cf3948f7bba1f413cd9b3a2af2e69336c1286b51aebe65f2aa8aeaadc1305f not found: ID does not exist" Aug 04 01:04:41.740113 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.740080 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ddf6e-865dd88-smxzt"] Aug 04 01:04:41.750490 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.750464 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96"] Aug 04 01:04:41.753955 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.753930 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5ddf6e-588697f594-pzd96"] Aug 04 01:04:41.763753 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.763724 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz"] Aug 04 01:04:41.765916 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:41.765891 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ddf6e-c57d797d7-v8mbz"] Aug 04 01:04:42.378502 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:42.378427 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44466c87-396b-4399-8d06-a7b1afccbebb" path="/var/lib/kubelet/pods/44466c87-396b-4399-8d06-a7b1afccbebb/volumes" Aug 04 01:04:42.378859 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:42.378746 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" path="/var/lib/kubelet/pods/72fe2a82-f6ba-4c6b-b5bb-624fa5337d97/volumes" Aug 04 01:04:42.379152 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:42.379131 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c03320-ffc2-4f6c-b128-872e4c506776" path="/var/lib/kubelet/pods/89c03320-ffc2-4f6c-b128-872e4c506776/volumes" Aug 04 01:04:42.379588 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:42.379571 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" path="/var/lib/kubelet/pods/a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb/volumes" Aug 04 01:04:42.380230 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:42.380214 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b091241a-3875-47f5-a6ba-958e47da69a0" path="/var/lib/kubelet/pods/b091241a-3875-47f5-a6ba-958e47da69a0/volumes" Aug 04 01:04:48.399502 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.399457 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz"] Aug 04 01:04:48.400025 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400005 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c03320-ffc2-4f6c-b128-872e4c506776" containerName="revoke-keys" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400028 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c03320-ffc2-4f6c-b128-872e4c506776" containerName="revoke-keys" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400057 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" containerName="maas-api" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400066 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" containerName="maas-api" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400091 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" containerName="payload-processing" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400116 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" containerName="payload-processing" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400129 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b091241a-3875-47f5-a6ba-958e47da69a0" containerName="payload-pre-processing" Aug 04 01:04:48.400138 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400138 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b091241a-3875-47f5-a6ba-958e47da69a0" containerName="payload-pre-processing" Aug 04 01:04:48.400504 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400244 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="72fe2a82-f6ba-4c6b-b5bb-624fa5337d97" containerName="payload-processing" Aug 04 01:04:48.400504 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400257 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b091241a-3875-47f5-a6ba-958e47da69a0" containerName="payload-pre-processing" Aug 04 01:04:48.400504 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400273 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34cc9e4-2c9a-4be3-b3ea-89f5ad3c6abb" containerName="maas-api" Aug 04 01:04:48.400504 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.400283 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c03320-ffc2-4f6c-b128-872e4c506776" containerName="revoke-keys" Aug 04 01:04:48.405381 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.405359 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:48.408038 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.407950 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgzb2\"" Aug 04 01:04:48.410168 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.410146 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz"] Aug 04 01:04:48.577505 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.577462 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644d136f-1d00-4eed-ae2b-f5d946dd5d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz\" (UID: \"644d136f-1d00-4eed-ae2b-f5d946dd5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:48.678928 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.678835 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644d136f-1d00-4eed-ae2b-f5d946dd5d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz\" (UID: \"644d136f-1d00-4eed-ae2b-f5d946dd5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:48.679554 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.679531 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644d136f-1d00-4eed-ae2b-f5d946dd5d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz\" (UID: \"644d136f-1d00-4eed-ae2b-f5d946dd5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:48.718045 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.718003 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:48.856995 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:48.856966 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz"] Aug 04 01:04:48.859178 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:04:48.859147 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644d136f_1d00_4eed_ae2b_f5d946dd5d9a.slice/crio-5c4fce354d283f125172af534005921b2b9a8004325ffb3b309c82b3797ee417 WatchSource:0}: Error finding container 5c4fce354d283f125172af534005921b2b9a8004325ffb3b309c82b3797ee417: Status 404 returned error can't find the container with id 5c4fce354d283f125172af534005921b2b9a8004325ffb3b309c82b3797ee417 Aug 04 01:04:49.767858 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:49.767813 2571 generic.go:356] "Generic (PLEG): container finished" podID="644d136f-1d00-4eed-ae2b-f5d946dd5d9a" containerID="fa0f04cdcbdf8c31ffeda5a9461346812b6173e4e445340fef6ef35df1263891" exitCode=0 Aug 04 01:04:49.768345 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:49.767912 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" event={"ID":"644d136f-1d00-4eed-ae2b-f5d946dd5d9a","Type":"ContainerDied","Data":"fa0f04cdcbdf8c31ffeda5a9461346812b6173e4e445340fef6ef35df1263891"} Aug 04 01:04:49.768345 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:49.768027 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" event={"ID":"644d136f-1d00-4eed-ae2b-f5d946dd5d9a","Type":"ContainerStarted","Data":"5c4fce354d283f125172af534005921b2b9a8004325ffb3b309c82b3797ee417"} Aug 04 01:04:50.906539 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:50.906511 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:51.002188 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:51.002152 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644d136f-1d00-4eed-ae2b-f5d946dd5d9a-maas-api-service-ca\") pod \"644d136f-1d00-4eed-ae2b-f5d946dd5d9a\" (UID: \"644d136f-1d00-4eed-ae2b-f5d946dd5d9a\") " Aug 04 01:04:51.002535 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:51.002508 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644d136f-1d00-4eed-ae2b-f5d946dd5d9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "644d136f-1d00-4eed-ae2b-f5d946dd5d9a" (UID: "644d136f-1d00-4eed-ae2b-f5d946dd5d9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:51.103601 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:51.103509 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/644d136f-1d00-4eed-ae2b-f5d946dd5d9a-maas-api-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:51.781126 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:51.781068 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" Aug 04 01:04:51.781323 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:51.781121 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz" event={"ID":"644d136f-1d00-4eed-ae2b-f5d946dd5d9a","Type":"ContainerDied","Data":"5c4fce354d283f125172af534005921b2b9a8004325ffb3b309c82b3797ee417"} Aug 04 01:04:51.781323 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:51.781179 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4fce354d283f125172af534005921b2b9a8004325ffb3b309c82b3797ee417" Aug 04 01:04:52.808878 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:52.808836 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c"] Aug 04 01:04:52.809571 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:52.809533 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" podUID="24448f5b-8b6e-4e35-8813-91d0a8cdfec5" containerName="maas-api" containerID="cri-o://9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f" gracePeriod=30 Aug 04 01:04:52.875961 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:52.875919 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc"] Aug 04 01:04:52.876265 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:52.876230 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" podUID="c82a60ef-4c0c-47d2-9d91-7add0369fb73" containerName="payload-processing" containerID="cri-o://91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07" gracePeriod=30 Aug 04 01:04:52.883572 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:52.883538 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j"] Aug 04 01:04:52.883856 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:52.883798 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" podUID="566208e5-a439-4941-9791-24014b3dea14" containerName="payload-pre-processing" containerID="cri-o://1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4" gracePeriod=30 Aug 04 01:04:53.174761 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.174718 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 01:04:53.177605 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.177579 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz"] Aug 04 01:04:53.181959 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.181925 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-33bda8-hwzgz"] Aug 04 01:04:53.185065 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.185047 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 01:04:53.191727 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.191707 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 01:04:53.324853 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.324819 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkqqq\" (UniqueName: \"kubernetes.io/projected/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-kube-api-access-jkqqq\") pod \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " Aug 04 01:04:53.324853 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.324859 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566208e5-a439-4941-9791-24014b3dea14-plugins-config-volume\") pod \"566208e5-a439-4941-9791-24014b3dea14\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " Aug 04 01:04:53.325097 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.324887 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls\") pod \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " Aug 04 01:04:53.325097 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325062 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/c82a60ef-4c0c-47d2-9d91-7add0369fb73-kube-api-access-s8rgq\") pod \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " Aug 04 01:04:53.325258 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325160 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74cj4\" (UniqueName: \"kubernetes.io/projected/566208e5-a439-4941-9791-24014b3dea14-kube-api-access-74cj4\") pod \"566208e5-a439-4941-9791-24014b3dea14\" (UID: \"566208e5-a439-4941-9791-24014b3dea14\") " Aug 04 01:04:53.325258 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325184 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82a60ef-4c0c-47d2-9d91-7add0369fb73-plugins-config-volume\") pod \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\" (UID: \"c82a60ef-4c0c-47d2-9d91-7add0369fb73\") " Aug 04 01:04:53.325258 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325202 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-openshift-service-ca\") pod \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\" (UID: \"24448f5b-8b6e-4e35-8813-91d0a8cdfec5\") " Aug 04 01:04:53.325400 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325299 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566208e5-a439-4941-9791-24014b3dea14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "566208e5-a439-4941-9791-24014b3dea14" (UID: "566208e5-a439-4941-9791-24014b3dea14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:53.325642 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325615 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24448f5b-8b6e-4e35-8813-91d0a8cdfec5" (UID: "24448f5b-8b6e-4e35-8813-91d0a8cdfec5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:53.325777 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325651 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82a60ef-4c0c-47d2-9d91-7add0369fb73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c82a60ef-4c0c-47d2-9d91-7add0369fb73" (UID: "c82a60ef-4c0c-47d2-9d91-7add0369fb73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 01:04:53.325777 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.325628 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566208e5-a439-4941-9791-24014b3dea14-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.327279 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.327247 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-kube-api-access-jkqqq" (OuterVolumeSpecName: "kube-api-access-jkqqq") pod "24448f5b-8b6e-4e35-8813-91d0a8cdfec5" (UID: "24448f5b-8b6e-4e35-8813-91d0a8cdfec5"). InnerVolumeSpecName "kube-api-access-jkqqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:53.327418 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.327395 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566208e5-a439-4941-9791-24014b3dea14-kube-api-access-74cj4" (OuterVolumeSpecName: "kube-api-access-74cj4") pod "566208e5-a439-4941-9791-24014b3dea14" (UID: "566208e5-a439-4941-9791-24014b3dea14"). InnerVolumeSpecName "kube-api-access-74cj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:53.327573 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.327552 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24448f5b-8b6e-4e35-8813-91d0a8cdfec5" (UID: "24448f5b-8b6e-4e35-8813-91d0a8cdfec5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 01:04:53.327573 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.327560 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82a60ef-4c0c-47d2-9d91-7add0369fb73-kube-api-access-s8rgq" (OuterVolumeSpecName: "kube-api-access-s8rgq") pod "c82a60ef-4c0c-47d2-9d91-7add0369fb73" (UID: "c82a60ef-4c0c-47d2-9d91-7add0369fb73"). InnerVolumeSpecName "kube-api-access-s8rgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 01:04:53.426519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.426421 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqqq\" (UniqueName: \"kubernetes.io/projected/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-kube-api-access-jkqqq\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.426519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.426453 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-maas-api-tls\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.426519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.426470 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/c82a60ef-4c0c-47d2-9d91-7add0369fb73-kube-api-access-s8rgq\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.426519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.426483 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74cj4\" (UniqueName: \"kubernetes.io/projected/566208e5-a439-4941-9791-24014b3dea14-kube-api-access-74cj4\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.426519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.426496 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82a60ef-4c0c-47d2-9d91-7add0369fb73-plugins-config-volume\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.426519 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.426507 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24448f5b-8b6e-4e35-8813-91d0a8cdfec5-openshift-service-ca\") on node \"ip-10-0-134-15.ec2.internal\" DevicePath \"\"" Aug 04 01:04:53.794400 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.794357 2571 generic.go:356] "Generic (PLEG): container finished" podID="c82a60ef-4c0c-47d2-9d91-7add0369fb73" containerID="91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07" exitCode=0 Aug 04 01:04:53.794721 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.794453 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" Aug 04 01:04:53.794721 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.794458 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" event={"ID":"c82a60ef-4c0c-47d2-9d91-7add0369fb73","Type":"ContainerDied","Data":"91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07"} Aug 04 01:04:53.794721 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.794566 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc" event={"ID":"c82a60ef-4c0c-47d2-9d91-7add0369fb73","Type":"ContainerDied","Data":"0e32dd759510d480ab9ceac353aa92af1c4dd8b88a817c64d69b4ddea8bc3d7d"} Aug 04 01:04:53.794721 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.794609 2571 scope.go:117] "RemoveContainer" containerID="91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07" Aug 04 01:04:53.796213 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.796177 2571 generic.go:356] "Generic (PLEG): container finished" podID="566208e5-a439-4941-9791-24014b3dea14" containerID="1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4" exitCode=0 Aug 04 01:04:53.796334 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.796244 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" Aug 04 01:04:53.796334 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.796274 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" event={"ID":"566208e5-a439-4941-9791-24014b3dea14","Type":"ContainerDied","Data":"1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4"} Aug 04 01:04:53.796334 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.796329 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j" event={"ID":"566208e5-a439-4941-9791-24014b3dea14","Type":"ContainerDied","Data":"a79ac66f1412b09d0b5711076b25c3baf04aa4b81046e33903d16e1f372ca8a7"} Aug 04 01:04:53.798734 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.798712 2571 generic.go:356] "Generic (PLEG): container finished" podID="24448f5b-8b6e-4e35-8813-91d0a8cdfec5" containerID="9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f" exitCode=0 Aug 04 01:04:53.798847 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.798786 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" Aug 04 01:04:53.798847 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.798813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" event={"ID":"24448f5b-8b6e-4e35-8813-91d0a8cdfec5","Type":"ContainerDied","Data":"9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f"} Aug 04 01:04:53.798970 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.798854 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c" event={"ID":"24448f5b-8b6e-4e35-8813-91d0a8cdfec5","Type":"ContainerDied","Data":"9f2cc25c03cf1b03e41ad23c39b76dbca9f699f7767bb44470f1480adbcc41ce"} Aug 04 01:04:53.805395 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.805370 2571 scope.go:117] "RemoveContainer" containerID="91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07" Aug 04 01:04:53.806267 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:04:53.806239 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07\": container with ID starting with 91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07 not found: ID does not exist" containerID="91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07" Aug 04 01:04:53.806376 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.806277 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07"} err="failed to get container status \"91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07\": rpc error: code = NotFound desc = could not find container \"91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07\": container with ID starting with 91b818abc9fb0528d8440035c2a5649e41dace6ee0fd33bafba195d61fe25b07 not found: ID does not exist" Aug 04 01:04:53.806376 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.806313 2571 scope.go:117] "RemoveContainer" containerID="1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4" Aug 04 01:04:53.816459 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.816319 2571 scope.go:117] "RemoveContainer" containerID="1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4" Aug 04 01:04:53.816661 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:04:53.816564 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4\": container with ID starting with 1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4 not found: ID does not exist" containerID="1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4" Aug 04 01:04:53.816661 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.816589 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4"} err="failed to get container status \"1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4\": rpc error: code = NotFound desc = could not find container \"1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4\": container with ID starting with 1ac11def4f423291dc487a0e8ab4705dd6a42f06bbd19d4b248167e147f8cdd4 not found: ID does not exist" Aug 04 01:04:53.816661 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.816606 2571 scope.go:117] "RemoveContainer" containerID="9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f" Aug 04 01:04:53.823054 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.823033 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc"] Aug 04 01:04:53.827645 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.827623 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-33bda8-69f8cbfbb6-l26kc"] Aug 04 01:04:53.837065 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.837035 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c"] Aug 04 01:04:53.839123 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.839069 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-33bda8-6d64599c5f-27w7c"] Aug 04 01:04:53.839199 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.839167 2571 scope.go:117] "RemoveContainer" containerID="9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f" Aug 04 01:04:53.839513 ip-10-0-134-15 kubenswrapper[2571]: E0804 01:04:53.839494 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f\": container with ID starting with 9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f not found: ID does not exist" containerID="9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f" Aug 04 01:04:53.839574 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.839519 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f"} err="failed to get container status \"9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f\": rpc error: code = NotFound desc = could not find container \"9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f\": container with ID starting with 9a4af72cc3de45df39be5d98bda4240247e1c6bebf79f5babe1446db04f65d1f not found: ID does not exist" Aug 04 01:04:53.847629 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.847594 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j"] Aug 04 01:04:53.849730 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:53.849695 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-33bda8-6d459b9c9b-d969j"] Aug 04 01:04:54.377003 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:54.376966 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24448f5b-8b6e-4e35-8813-91d0a8cdfec5" path="/var/lib/kubelet/pods/24448f5b-8b6e-4e35-8813-91d0a8cdfec5/volumes" Aug 04 01:04:54.377350 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:54.377337 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566208e5-a439-4941-9791-24014b3dea14" path="/var/lib/kubelet/pods/566208e5-a439-4941-9791-24014b3dea14/volumes" Aug 04 01:04:54.377641 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:54.377629 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644d136f-1d00-4eed-ae2b-f5d946dd5d9a" path="/var/lib/kubelet/pods/644d136f-1d00-4eed-ae2b-f5d946dd5d9a/volumes" Aug 04 01:04:54.377937 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:04:54.377926 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c82a60ef-4c0c-47d2-9d91-7add0369fb73" path="/var/lib/kubelet/pods/c82a60ef-4c0c-47d2-9d91-7add0369fb73/volumes" Aug 04 01:05:06.228601 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:06.228569 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b6cbdb5f9-kgpcj_6805a8f3-adea-4c35-8ef8-c41d0e80b1e5/manager/0.log" Aug 04 01:05:06.870027 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:06.869998 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-xn65n_dc2050fa-6d9b-4e19-b71a-5fdae40301e4/copy-manifests/0.log" Aug 04 01:05:06.877279 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:06.877243 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-xn65n_dc2050fa-6d9b-4e19-b71a-5fdae40301e4/manager/0.log" Aug 04 01:05:07.270759 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:07.270729 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67c8968587-lpv6f_9b0d62de-09ea-471d-8c61-b0b75b1afc87/maas-api/0.log" Aug 04 01:05:07.501562 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:07.501537 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29763405-njwfh_0b444cfe-8337-44fd-a731-7f8eadc8fb38/cleanup/0.log" Aug 04 01:05:07.740466 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:07.740380 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lffqj_78548bf1-7c9c-41b2-b775-ae06a31f43a4/postgres/0.log" Aug 04 01:05:09.856241 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:09.856209 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jhnq5_53eb014e-72e1-47af-968e-602fc81ab637/discovery/0.log" Aug 04 01:05:10.338405 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:10.338372 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-njlwl_6d43038d-b43d-48ec-920e-7df1ddc144d6/payload-pre-processing/0.log" Aug 04 01:05:10.580476 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:10.580445 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56c65ccbd6-nsg9s_272f48c6-278a-409c-a1e5-a300dcb4a2aa/router/0.log" Aug 04 01:05:10.957746 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:10.957658 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-k25h4_fb1c8d19-9577-44dc-8f21-b23640984439/main/0.log" Aug 04 01:05:11.309244 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:11.309197 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4gksr_50b80ba0-4de2-42d8-b1c6-02153a91be21/main/0.log" Aug 04 01:05:19.897339 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:19.897308 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sbgfn_8ae9a103-10de-4451-bb24-97976795b3c5/global-pull-secret-syncer/0.log" Aug 04 01:05:20.047747 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:20.047711 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s66z8_116551c0-6d82-4e9e-be71-7a11d2ecf733/konnectivity-agent/0.log" Aug 04 01:05:20.092804 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:20.092772 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-15.ec2.internal_b91cd527bb73e98102acd9e9c144cdda/haproxy/0.log" Aug 04 01:05:27.886202 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:27.886168 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dc9c95f58-cr2nt_c978303d-125e-433b-8cae-6532621f590d/metrics-server/0.log" Aug 04 01:05:27.946827 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:27.946795 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qb9dn_9a6e8357-0aec-4b62-906a-e471bf0e256c/node-exporter/0.log" Aug 04 01:05:27.965218 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:27.965184 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qb9dn_9a6e8357-0aec-4b62-906a-e471bf0e256c/kube-rbac-proxy/0.log" Aug 04 01:05:27.984313 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:27.984285 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qb9dn_9a6e8357-0aec-4b62-906a-e471bf0e256c/init-textfile/0.log" Aug 04 01:05:28.153886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.153803 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-zwbsn_501ab213-04bd-48c1-be3c-96f343f8b601/kube-rbac-proxy-main/0.log" Aug 04 01:05:28.174799 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.174757 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-zwbsn_501ab213-04bd-48c1-be3c-96f343f8b601/kube-rbac-proxy-self/0.log" Aug 04 01:05:28.195148 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.195115 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-zwbsn_501ab213-04bd-48c1-be3c-96f343f8b601/openshift-state-metrics/0.log" Aug 04 01:05:28.563129 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.563085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc5d646c7-fg787_752f8113-f595-4dbb-8367-e3573e724f8a/thanos-query/0.log" Aug 04 01:05:28.584900 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.584871 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc5d646c7-fg787_752f8113-f595-4dbb-8367-e3573e724f8a/kube-rbac-proxy-web/0.log" Aug 04 01:05:28.606044 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.606012 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc5d646c7-fg787_752f8113-f595-4dbb-8367-e3573e724f8a/kube-rbac-proxy/0.log" Aug 04 01:05:28.625735 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.625706 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc5d646c7-fg787_752f8113-f595-4dbb-8367-e3573e724f8a/prom-label-proxy/0.log" Aug 04 01:05:28.645948 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.645922 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc5d646c7-fg787_752f8113-f595-4dbb-8367-e3573e724f8a/kube-rbac-proxy-rules/0.log" Aug 04 01:05:28.667796 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:28.667769 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc5d646c7-fg787_752f8113-f595-4dbb-8367-e3573e724f8a/kube-rbac-proxy-metrics/0.log" Aug 04 01:05:29.764634 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:29.764581 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-whqmh_788bda68-1682-402e-8cea-39859d7323e3/networking-console-plugin/0.log" Aug 04 01:05:30.000645 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.000608 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc"] Aug 04 01:05:30.001093 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001073 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d136f-1d00-4eed-ae2b-f5d946dd5d9a" containerName="revoke-keys" Aug 04 01:05:30.001093 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001096 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d136f-1d00-4eed-ae2b-f5d946dd5d9a" containerName="revoke-keys" Aug 04 01:05:30.001282 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001140 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24448f5b-8b6e-4e35-8813-91d0a8cdfec5" containerName="maas-api" Aug 04 01:05:30.001282 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001148 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24448f5b-8b6e-4e35-8813-91d0a8cdfec5" containerName="maas-api" Aug 04 01:05:30.001282 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001183 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566208e5-a439-4941-9791-24014b3dea14" containerName="payload-pre-processing" Aug 04 01:05:30.001282 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001192 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="566208e5-a439-4941-9791-24014b3dea14" containerName="payload-pre-processing" Aug 04 01:05:30.001282 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001220 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82a60ef-4c0c-47d2-9d91-7add0369fb73" containerName="payload-processing" Aug 04 01:05:30.001282 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001230 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82a60ef-4c0c-47d2-9d91-7add0369fb73" containerName="payload-processing" Aug 04 01:05:30.001594 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001329 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="566208e5-a439-4941-9791-24014b3dea14" containerName="payload-pre-processing" Aug 04 01:05:30.001594 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001345 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82a60ef-4c0c-47d2-9d91-7add0369fb73" containerName="payload-processing" Aug 04 01:05:30.001594 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001356 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="644d136f-1d00-4eed-ae2b-f5d946dd5d9a" containerName="revoke-keys" Aug 04 01:05:30.001594 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.001367 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24448f5b-8b6e-4e35-8813-91d0a8cdfec5" containerName="maas-api" Aug 04 01:05:30.004184 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.004164 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.006446 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.006423 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cdcm\"/\"openshift-service-ca.crt\"" Aug 04 01:05:30.007395 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.007375 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9cdcm\"/\"default-dockercfg-wr28w\"" Aug 04 01:05:30.007652 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.007635 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9cdcm\"/\"kube-root-ca.crt\"" Aug 04 01:05:30.011116 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.011075 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc"] Aug 04 01:05:30.144791 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.144732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-proc\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.144791 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.144790 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-sys\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.145046 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.144812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-lib-modules\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.145046 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.144869 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvlkc\" (UniqueName: \"kubernetes.io/projected/26099bdf-07c5-4809-b250-490e1d55fd05-kube-api-access-gvlkc\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.145046 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.144890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-podres\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245638 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245603 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-proc\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245638 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-sys\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-lib-modules\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245718 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvlkc\" (UniqueName: \"kubernetes.io/projected/26099bdf-07c5-4809-b250-490e1d55fd05-kube-api-access-gvlkc\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245729 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-proc\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245737 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-sys\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-podres\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245821 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-podres\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.245886 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.245840 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26099bdf-07c5-4809-b250-490e1d55fd05-lib-modules\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.253564 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.253540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvlkc\" (UniqueName: \"kubernetes.io/projected/26099bdf-07c5-4809-b250-490e1d55fd05-kube-api-access-gvlkc\") pod \"perf-node-gather-daemonset-ffbmc\" (UID: \"26099bdf-07c5-4809-b250-490e1d55fd05\") " pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.315640 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.315602 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:30.446791 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.446635 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc"] Aug 04 01:05:30.449162 ip-10-0-134-15 kubenswrapper[2571]: W0804 01:05:30.449137 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26099bdf_07c5_4809_b250_490e1d55fd05.slice/crio-a79beef8c872bd8a315cd1339e2a10833cec964ed423a7ff53d447924d3499e3 WatchSource:0}: Error finding container a79beef8c872bd8a315cd1339e2a10833cec964ed423a7ff53d447924d3499e3: Status 404 returned error can't find the container with id a79beef8c872bd8a315cd1339e2a10833cec964ed423a7ff53d447924d3499e3 Aug 04 01:05:30.966168 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:30.966088 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-558f4699d9-lgczg_992df4df-7120-4af9-bd77-043471e6ba41/console/0.log" Aug 04 01:05:31.009355 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:31.009303 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" event={"ID":"26099bdf-07c5-4809-b250-490e1d55fd05","Type":"ContainerStarted","Data":"324ad56267c66f7d89805432eb587ac857668174bf11624a54b20cc164d52381"} Aug 04 01:05:31.009355 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:31.009359 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" event={"ID":"26099bdf-07c5-4809-b250-490e1d55fd05","Type":"ContainerStarted","Data":"a79beef8c872bd8a315cd1339e2a10833cec964ed423a7ff53d447924d3499e3"} Aug 04 01:05:31.009676 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:31.009391 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:31.009926 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:31.009905 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-44zf4_af40b08d-2067-4c24-b2ad-d53d0e015a37/download-server/0.log" Aug 04 01:05:31.025702 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:31.025656 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" podStartSLOduration=2.025641411 podStartE2EDuration="2.025641411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 01:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 01:05:31.023362879 +0000 UTC m=+3769.194396186" watchObservedRunningTime="2026-08-04 01:05:31.025641411 +0000 UTC m=+3769.196674755" Aug 04 01:05:32.663173 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:32.663143 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-htm5p_277504f0-4353-4cca-9cc2-3610dbdd9fdc/dns/0.log" Aug 04 01:05:32.684946 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:32.684918 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-htm5p_277504f0-4353-4cca-9cc2-3610dbdd9fdc/kube-rbac-proxy/0.log" Aug 04 01:05:32.705940 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:32.705913 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kx9jg_539e58ad-eeb7-4a35-8d3f-af0cebbbe186/dns-node-resolver/0.log" Aug 04 01:05:33.380056 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:33.380024 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tpvjb_f95614fb-0969-4b83-9db8-7bbc91885d23/node-ca/0.log" Aug 04 01:05:34.626887 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:34.626802 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jhnq5_53eb014e-72e1-47af-968e-602fc81ab637/discovery/0.log" Aug 04 01:05:34.757155 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:34.757126 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-njlwl_6d43038d-b43d-48ec-920e-7df1ddc144d6/payload-pre-processing/0.log" Aug 04 01:05:34.813825 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:34.813788 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56c65ccbd6-nsg9s_272f48c6-278a-409c-a1e5-a300dcb4a2aa/router/0.log" Aug 04 01:05:35.612168 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:35.612132 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lb2n9_9b09fb90-1aef-4889-8783-bd14933ff063/serve-healthcheck-canary/0.log" Aug 04 01:05:36.254888 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:36.254861 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zw2f8_968e1233-5b75-459b-9f7c-4ca46be27371/kube-rbac-proxy/0.log" Aug 04 01:05:36.275718 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:36.275692 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zw2f8_968e1233-5b75-459b-9f7c-4ca46be27371/exporter/0.log" Aug 04 01:05:36.295111 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:36.295084 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zw2f8_968e1233-5b75-459b-9f7c-4ca46be27371/extractor/0.log" Aug 04 01:05:37.024866 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:37.024838 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9cdcm/perf-node-gather-daemonset-ffbmc" Aug 04 01:05:39.170943 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:39.170867 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b6cbdb5f9-kgpcj_6805a8f3-adea-4c35-8ef8-c41d0e80b1e5/manager/0.log" Aug 04 01:05:39.613425 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:39.613391 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-xn65n_dc2050fa-6d9b-4e19-b71a-5fdae40301e4/manager/0.log" Aug 04 01:05:39.636330 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:39.636305 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b98d597c-xn65n_dc2050fa-6d9b-4e19-b71a-5fdae40301e4/copy-manifests/0.log" Aug 04 01:05:40.556785 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:40.556745 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jhnq5_53eb014e-72e1-47af-968e-602fc81ab637/discovery/0.log" Aug 04 01:05:40.637910 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:40.637863 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-njlwl_6d43038d-b43d-48ec-920e-7df1ddc144d6/payload-pre-processing/0.log" Aug 04 01:05:40.679056 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:40.679023 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56c65ccbd6-nsg9s_272f48c6-278a-409c-a1e5-a300dcb4a2aa/router/0.log" Aug 04 01:05:40.872000 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:40.871909 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cfb669bf4-5kw7h_fe785b11-fa5f-453d-ada5-29e8ce2dca2a/manager/0.log" Aug 04 01:05:42.460875 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:42.460838 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jhnq5_53eb014e-72e1-47af-968e-602fc81ab637/discovery/0.log" Aug 04 01:05:42.546859 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:42.546824 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-njlwl_6d43038d-b43d-48ec-920e-7df1ddc144d6/payload-pre-processing/0.log" Aug 04 01:05:42.593604 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:42.593575 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56c65ccbd6-nsg9s_272f48c6-278a-409c-a1e5-a300dcb4a2aa/router/0.log" Aug 04 01:05:45.708646 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:45.708588 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-qxvlm_cf63f97c-c35a-4b5d-978a-d6178e0c45f8/migrator/0.log" Aug 04 01:05:45.726774 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:45.726746 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-qxvlm_cf63f97c-c35a-4b5d-978a-d6178e0c45f8/graceful-termination/0.log" Aug 04 01:05:46.042623 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:46.042594 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-z2hl8_a2be9d4e-c830-49a6-91fe-b1d98984dd98/kube-storage-version-migrator-operator/1.log" Aug 04 01:05:46.044458 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:46.044433 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-z2hl8_a2be9d4e-c830-49a6-91fe-b1d98984dd98/kube-storage-version-migrator-operator/0.log" Aug 04 01:05:47.059009 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.058973 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6t2wv_051a04fe-978e-48ea-878f-71bf7c22419e/kube-multus/0.log" Aug 04 01:05:47.082811 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.082779 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/kube-multus-additional-cni-plugins/0.log" Aug 04 01:05:47.102459 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.102429 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/egress-router-binary-copy/0.log" Aug 04 01:05:47.122363 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.122339 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/cni-plugins/0.log" Aug 04 01:05:47.141198 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.141158 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/bond-cni-plugin/0.log" Aug 04 01:05:47.162515 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.162484 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/routeoverride-cni/0.log" Aug 04 01:05:47.184653 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.184630 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/whereabouts-cni-bincopy/0.log" Aug 04 01:05:47.204521 ip-10-0-134-15 kubenswrapper[2571]: I0804 01:05:47.204492 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-46gf6_e000d8a9-3ea5-4106-96cc-2a8b36c1a680/whereabouts-cni/0.log"