Jul 19 07:42:32.101975 ip-10-0-132-126 systemd[1]: Starting Kubernetes Kubelet... Jul 19 07:42:32.497747 ip-10-0-132-126 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 07:42:32.497747 ip-10-0-132-126 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 19 07:42:32.497747 ip-10-0-132-126 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 07:42:32.497747 ip-10-0-132-126 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 19 07:42:32.497747 ip-10-0-132-126 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 07:42:32.498717 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.498525 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 19 07:42:32.502540 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502522 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:32.502540 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502540 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:32.502540 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502544 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502547 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502550 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502554 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502557 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502560 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502562 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502565 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502568 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502570 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502573 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502582 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502585 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502588 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502590 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502593 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502596 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502598 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502601 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:32.502655 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502605 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502609 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502613 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502616 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502619 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502623 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502626 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502628 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502631 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502634 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502637 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502640 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502644 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502647 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502650 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502653 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502656 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502659 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502662 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:32.503116 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502664 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502667 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502670 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502686 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502689 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502692 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502694 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502697 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502700 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502702 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502705 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502707 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502710 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502713 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502715 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502718 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502721 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502723 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502726 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502728 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:32.503574 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502730 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502733 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502735 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502738 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502740 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502743 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502745 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502748 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502752 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502755 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502758 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502760 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502763 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502765 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502768 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502770 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502773 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502775 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502777 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:32.504121 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502780 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502782 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502785 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502788 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502791 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502793 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502795 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502798 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502801 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502803 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502806 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502808 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502811 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502814 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502816 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502818 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502821 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502823 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502825 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.502828 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:32.504576 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503529 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503535 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503539 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503545 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503549 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503551 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503554 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503556 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503559 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503561 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503564 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503566 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503569 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503571 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503574 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503578 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503581 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503584 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503587 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:32.505085 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503589 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503592 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503594 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503597 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503599 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503602 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503606 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503609 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503612 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503615 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503618 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503620 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503623 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503625 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503628 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503631 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503634 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503637 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503640 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503642 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:32.505546 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503646 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503649 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503651 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503653 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503656 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503659 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503661 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503665 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503669 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503671 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503692 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503695 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503697 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503700 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503702 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503705 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503708 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503710 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503712 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:32.506055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503715 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503717 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503720 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503723 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503726 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503728 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503730 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503734 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503737 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503739 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503742 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503745 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503748 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503750 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503753 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503756 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503758 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503761 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503763 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503766 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:32.506519 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503768 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503770 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503773 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503775 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503778 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503780 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503783 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503785 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503788 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503790 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503793 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503795 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503797 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503800 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503802 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503804 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503807 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503809 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503812 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503815 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:32.507023 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.503818 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504202 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504208 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504212 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504215 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504217 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504220 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504223 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504226 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504228 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504231 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504233 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504236 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504238 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504241 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504243 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504246 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504248 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504251 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504253 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:32.507501 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504255 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504258 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504261 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504263 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504265 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504268 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504271 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504273 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504276 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504278 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504280 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504286 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504288 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504291 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504293 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504296 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504299 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504301 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504304 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504307 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:32.508011 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504309 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504312 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504314 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504317 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504322 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504325 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504343 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504347 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504352 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504355 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504358 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504361 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504363 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504366 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504369 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504371 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504374 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504377 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504380 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:32.508485 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504382 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504385 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504387 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504390 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504394 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504396 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504399 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504401 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504405 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504409 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504412 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504415 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504418 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504420 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504423 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504426 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504430 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504434 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504437 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:32.509008 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504439 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504442 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504446 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504448 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504451 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504453 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504456 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504458 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504461 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504464 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504466 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504469 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504472 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504474 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504477 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504479 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504481 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504485 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504488 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504490 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:32.509471 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504492 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.504495 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504564 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504571 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504577 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504581 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504586 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504589 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504593 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504597 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504600 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504603 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504606 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504609 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504613 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504616 2574 flags.go:64] FLAG: --cgroup-root="" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504619 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504622 2574 flags.go:64] FLAG: --client-ca-file="" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504625 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504628 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504632 2574 flags.go:64] FLAG: --cluster-domain="" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504635 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504638 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504641 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 19 07:42:32.509978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504644 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504648 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504651 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504654 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504657 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504661 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504664 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504666 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504669 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504687 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504691 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504694 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504696 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504711 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504715 2574 flags.go:64] FLAG: --enable-server="true" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504719 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504723 2574 flags.go:64] FLAG: --event-burst="100" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504726 2574 flags.go:64] FLAG: --event-qps="50" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504729 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504732 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504735 2574 flags.go:64] FLAG: --eviction-hard="" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504739 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504742 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504745 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504748 2574 flags.go:64] FLAG: --eviction-soft="" Jul 19 07:42:32.510554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504751 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504753 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504757 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504760 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504763 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504766 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504768 2574 flags.go:64] FLAG: --feature-gates="" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504772 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504775 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504778 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504781 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504784 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504787 2574 flags.go:64] FLAG: --help="false" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504799 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504802 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504805 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504808 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504812 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504815 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504818 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504821 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504824 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504827 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 19 07:42:32.511160 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504830 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504833 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504836 2574 flags.go:64] FLAG: --kube-reserved="" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504838 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504841 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504844 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504847 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504850 2574 flags.go:64] FLAG: --lock-file="" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504852 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504855 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504859 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504864 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504867 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504869 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504872 2574 flags.go:64] FLAG: --logging-format="text" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504875 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504878 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504881 2574 flags.go:64] FLAG: --manifest-url="" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504883 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504887 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504890 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504894 2574 flags.go:64] FLAG: --max-pods="110" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504897 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504900 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504903 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 19 07:42:32.512155 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504906 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504909 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504912 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504914 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504921 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504924 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504927 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504930 2574 flags.go:64] FLAG: --pod-cidr="" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504933 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504938 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504940 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504943 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504946 2574 flags.go:64] FLAG: --port="10250" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504949 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504952 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088f140ef3f99e2b3" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504955 2574 flags.go:64] FLAG: --qos-reserved="" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504957 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504960 2574 flags.go:64] FLAG: --register-node="true" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504963 2574 flags.go:64] FLAG: --register-with-taints="" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504966 2574 flags.go:64] FLAG: --registry-burst="10" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504969 2574 flags.go:64] FLAG: --registry-qps="5" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504972 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504974 2574 flags.go:64] FLAG: --reserved-memory="" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504978 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504981 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 19 07:42:32.513297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504984 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504986 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504989 2574 flags.go:64] FLAG: --runonce="false" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504991 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504994 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.504997 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505000 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505002 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505005 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505008 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505011 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505014 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505016 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505019 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505022 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505025 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505028 2574 flags.go:64] FLAG: --system-cgroups="" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505030 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505035 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505038 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505042 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505046 2574 flags.go:64] FLAG: --tls-min-version="" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505049 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505051 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505054 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 19 07:42:32.514285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505057 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505060 2574 flags.go:64] FLAG: --v="2" Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505064 2574 flags.go:64] FLAG: --version="false" Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505067 2574 flags.go:64] FLAG: --vmodule="" Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505071 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.505075 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505192 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505196 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505199 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505202 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505205 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505209 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505212 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505214 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505217 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505219 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505222 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505225 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505227 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505230 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505232 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:32.515371 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505235 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505238 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505241 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505244 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505246 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505249 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505254 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505258 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505261 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505264 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505267 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505269 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505271 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505274 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505276 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505279 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505281 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505284 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505286 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:32.515940 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505288 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505291 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505293 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505297 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505299 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505302 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505305 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505307 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505309 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505312 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505314 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505317 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505319 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505322 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505324 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505327 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505330 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505332 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505334 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505338 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:32.516408 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505340 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505344 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505347 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505350 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505353 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505356 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505360 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505362 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505365 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505367 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505370 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505372 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505375 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505377 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505380 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505384 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505387 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505389 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505392 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505394 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:32.516916 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505397 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505399 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505402 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505404 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505407 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505409 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505412 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505414 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505417 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505420 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505422 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505427 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505429 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505432 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505434 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505437 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505440 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505442 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505444 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505447 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:32.517405 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505450 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505452 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505455 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505457 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.505460 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.506070 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.515824 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.515843 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.515857 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.515866 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515925 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515931 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515935 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515938 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515941 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:32.517985 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515944 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515946 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515949 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515952 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515955 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515957 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515960 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515963 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515966 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515968 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515971 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515973 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515976 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515988 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515991 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515995 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.515999 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516002 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516005 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:32.518368 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516009 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516014 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516016 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516020 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516023 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516027 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516029 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516032 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516035 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516038 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516041 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516044 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516046 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516049 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516052 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516054 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516057 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516060 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516062 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:32.518844 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516065 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516067 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516070 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516072 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516075 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516078 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516080 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516083 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516085 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516088 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516090 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516093 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516095 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516097 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516100 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516102 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516106 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516109 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516112 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516115 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:32.519302 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516118 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516120 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516123 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516125 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516128 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516130 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516133 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516135 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516137 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516141 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516145 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516147 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516150 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516153 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516156 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516158 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516163 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516165 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516168 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516170 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:32.519795 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516174 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516176 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516179 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516181 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516184 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516186 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516189 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516191 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516194 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516197 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516199 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516202 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516205 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516207 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516210 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:32.520274 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516212 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.516218 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516331 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516337 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516340 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516343 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516346 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516349 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516352 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516355 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516358 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516361 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516364 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516366 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516370 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 07:42:32.520651 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516373 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516376 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516379 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516382 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516385 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516388 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516390 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516392 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516395 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516397 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516400 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516402 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516405 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516408 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516411 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516413 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516417 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516419 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516422 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516425 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 07:42:32.521055 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516427 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516430 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516432 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516435 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516437 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516439 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516442 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516444 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516447 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516449 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516452 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516454 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516457 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516459 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516462 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516464 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516467 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516469 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516472 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516474 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 07:42:32.521542 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516477 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516479 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516482 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516484 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516487 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516490 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516492 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516496 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516500 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516502 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516505 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516508 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516510 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516513 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516516 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516518 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516520 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516523 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516525 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516528 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 07:42:32.522027 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516530 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516532 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516536 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516539 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516542 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516545 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516547 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516550 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516552 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516555 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516558 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516562 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516565 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516567 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516570 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516574 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516577 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516580 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 07:42:32.522504 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516583 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516585 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516588 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516590 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516593 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516595 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516598 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:32.516600 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.516605 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.516749 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.518868 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.522008 2574 server.go:1019] "Starting client certificate rotation" Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.522108 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 07:42:32.523000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.522139 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 07:42:32.528688 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.528661 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 07:42:32.533431 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.533414 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 07:42:32.545062 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.545043 2574 log.go:25] "Validated CRI v1 runtime API" Jul 19 07:42:32.547076 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.547060 2574 log.go:25] "Validated CRI v1 image API" Jul 19 07:42:32.547879 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.547865 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 19 07:42:32.551724 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.551699 2574 fs.go:135] Filesystem UUIDs: map[1db63360-7522-4440-8446-e6ec99e20e2a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cd353357-d176-48e3-b769-8852bc820d98:/dev/nvme0n1p4] Jul 19 07:42:32.551782 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.551726 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 19 07:42:32.557384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.557362 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 07:42:32.558689 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.558555 2574 manager.go:216] Machine: {Timestamp:2026-07-19 07:42:32.556589926 +0000 UTC m=+0.336253231 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100793 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c1fd6fac9245e191cb8f01e7b70a0 SystemUUID:ec2c1fd6-fac9-245e-191c-b8f01e7b70a0 BootID:0b1efa38-8e4f-499b-bde0-11c3bf2170cf Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:7c:0e:c5:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:7c:0e:c5:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:e5:29:b4:12:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 19 07:42:32.558689 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.558688 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 19 07:42:32.558804 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.558787 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 19 07:42:32.562064 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.561861 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 19 07:42:32.562213 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.562068 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 19 07:42:32.562696 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.562672 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 19 07:42:32.562737 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.562698 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 19 07:42:32.562737 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.562706 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 07:42:32.563271 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.563261 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 07:42:32.564372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.564362 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 19 07:42:32.567785 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.567772 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 19 07:42:32.567915 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.567905 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 19 07:42:32.567963 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.567955 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 19 07:42:32.568001 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.567968 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 19 07:42:32.568001 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.567982 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 19 07:42:32.568001 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.567991 2574 kubelet.go:398] "Adding apiserver pod source" Jul 19 07:42:32.568001 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.568001 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 19 07:42:32.568385 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.568372 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 07:42:32.568425 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.568397 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 07:42:32.570580 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.570558 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 19 07:42:32.572345 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.572331 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 19 07:42:32.572397 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.572362 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 19 07:42:32.573483 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573471 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 19 07:42:32.573519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573488 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 19 07:42:32.573519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573495 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 19 07:42:32.573519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573499 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 19 07:42:32.573519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573508 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 19 07:42:32.573519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573513 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 19 07:42:32.573646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573523 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 19 07:42:32.573646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573528 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 19 07:42:32.573646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573535 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 19 07:42:32.573646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573540 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 19 07:42:32.573646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573553 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 19 07:42:32.573646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.573558 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 19 07:42:32.574791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.574780 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 19 07:42:32.574791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.574790 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 19 07:42:32.577928 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.577913 2574 server.go:1268] "Started kubelet" Jul 19 07:42:32.578078 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.578019 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 19 07:42:32.578135 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.578107 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 19 07:42:32.578248 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.578216 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 19 07:42:32.578855 ip-10-0-132-126 systemd[1]: Started Kubernetes Kubelet. Jul 19 07:42:32.580401 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.580378 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 19 07:42:32.580497 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.580425 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 19 07:42:32.580646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.580632 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 19 07:42:32.580904 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.580884 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 19 07:42:32.581337 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.581273 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdlt8" Jul 19 07:42:32.583206 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.583184 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 19 07:42:32.585222 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.585205 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 19 07:42:32.585435 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.584584 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-126.ec2.internal.18c3a10c88eee0bc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-126.ec2.internal,UID:ip-10-0-132-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-126.ec2.internal,},FirstTimestamp:2026-07-19 07:42:32.577884348 +0000 UTC m=+0.357547653,LastTimestamp:2026-07-19 07:42:32.577884348 +0000 UTC m=+0.357547653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-126.ec2.internal,}" Jul 19 07:42:32.588586 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.588565 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 19 07:42:32.588693 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.588568 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 19 07:42:32.588764 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.588727 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:32.589500 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.589480 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 19 07:42:32.589600 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.589519 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 19 07:42:32.589600 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.589535 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 19 07:42:32.589600 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.589588 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdlt8" Jul 19 07:42:32.590463 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590339 2574 factory.go:153] Registering CRI-O factory Jul 19 07:42:32.590463 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590356 2574 factory.go:223] Registration of the crio container factory successfully Jul 19 07:42:32.590463 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590421 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 19 07:42:32.590463 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590432 2574 factory.go:55] Registering systemd factory Jul 19 07:42:32.590463 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590441 2574 factory.go:223] Registration of the systemd container factory successfully Jul 19 07:42:32.590749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590475 2574 factory.go:103] Registering Raw factory Jul 19 07:42:32.590749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.590489 2574 manager.go:1196] Started watching for new ooms in manager Jul 19 07:42:32.591199 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.591178 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 19 07:42:32.591319 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.591303 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 19 07:42:32.591509 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.591489 2574 manager.go:349] Starting recovery of all containers Jul 19 07:42:32.592782 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.592753 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 19 07:42:32.592782 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.592771 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 19 07:42:32.605049 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.605028 2574 manager.go:354] Recovery completed Jul 19 07:42:32.609047 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.609029 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.610791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.610772 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.610871 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.610803 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.610871 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.610814 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.611440 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.611425 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 19 07:42:32.611440 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.611439 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.611569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.611439 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 19 07:42:32.611569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.611468 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 19 07:42:32.612222 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.612206 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.612274 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.612236 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.612274 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.612247 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.613615 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.613484 2574 policy_none.go:49] "None policy: Start" Jul 19 07:42:32.613699 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.613621 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 19 07:42:32.613699 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.613635 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 19 07:42:32.615608 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.615593 2574 policy_none.go:47] "Start" Jul 19 07:42:32.660826 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.660799 2574 manager.go:337] "Starting Device Plugin manager" Jul 19 07:42:32.660973 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.660857 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 19 07:42:32.660973 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.660870 2574 server.go:87] "Starting device plugin registration server" Jul 19 07:42:32.661178 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.661164 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 19 07:42:32.661239 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.661178 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 19 07:42:32.661305 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.661272 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 19 07:42:32.661426 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.661369 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 19 07:42:32.661426 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.661379 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 19 07:42:32.661996 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.661972 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 19 07:42:32.662084 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.662033 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:32.724162 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.724128 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 19 07:42:32.726374 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.725536 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 19 07:42:32.726374 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.725570 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 19 07:42:32.726374 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.725596 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 19 07:42:32.726374 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.725628 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 19 07:42:32.728083 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.728061 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:32.762338 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.762258 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.763804 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.763788 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.763873 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.763820 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.763873 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.763832 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.763873 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.763858 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.772590 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.772557 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.772590 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.772587 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-126.ec2.internal\": node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:32.786351 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.786327 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:32.826399 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.826354 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal"] Jul 19 07:42:32.826587 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.826434 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.827393 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.827373 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.827476 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.827405 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.827476 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.827415 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.829583 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.829569 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.829712 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.829698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.829751 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.829728 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.831069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.831051 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.831133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.831087 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.831133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.831098 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.832707 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.832692 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.832756 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.832719 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.832756 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.832730 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.833193 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.833175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.833285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.833207 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 07:42:32.833911 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.833896 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientMemory" Jul 19 07:42:32.834004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.833929 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 07:42:32.834004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.833944 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeHasSufficientPID" Jul 19 07:42:32.862360 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.862328 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-126.ec2.internal\" not found" node="ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.866704 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.866658 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-126.ec2.internal\" not found" node="ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.886874 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.886849 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:32.894756 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.894736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/235d857ab46d455762737cb4a14294eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal\" (UID: \"235d857ab46d455762737cb4a14294eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.894848 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.894762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a13e305d78555894578d5848a397f831-config\") pod \"kube-apiserver-proxy-ip-10-0-132-126.ec2.internal\" (UID: \"a13e305d78555894578d5848a397f831\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.894848 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.894778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/235d857ab46d455762737cb4a14294eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal\" (UID: \"235d857ab46d455762737cb4a14294eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.987337 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:32.987306 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:32.995759 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.995732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a13e305d78555894578d5848a397f831-config\") pod \"kube-apiserver-proxy-ip-10-0-132-126.ec2.internal\" (UID: \"a13e305d78555894578d5848a397f831\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.995818 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.995764 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/235d857ab46d455762737cb4a14294eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal\" (UID: \"235d857ab46d455762737cb4a14294eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.995818 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.995784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/235d857ab46d455762737cb4a14294eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal\" (UID: \"235d857ab46d455762737cb4a14294eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.995879 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.995826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a13e305d78555894578d5848a397f831-config\") pod \"kube-apiserver-proxy-ip-10-0-132-126.ec2.internal\" (UID: \"a13e305d78555894578d5848a397f831\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.995879 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.995838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/235d857ab46d455762737cb4a14294eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal\" (UID: \"235d857ab46d455762737cb4a14294eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:32.995879 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:32.995849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/235d857ab46d455762737cb4a14294eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal\" (UID: \"235d857ab46d455762737cb4a14294eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:33.088264 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.088185 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.165746 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.165717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:33.169199 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.169181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" Jul 19 07:42:33.189172 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.189145 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.290172 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.290121 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.390313 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.390220 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.491200 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.491157 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.519643 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.519612 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 19 07:42:33.520140 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.519782 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 07:42:33.591831 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.591802 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.595464 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.595430 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 07:37:33 +0000 UTC" deadline="2027-12-23 04:54:00.412996688 +0000 UTC" Jul 19 07:42:33.595464 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.595458 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12525h11m26.817542429s" Jul 19 07:42:33.692980 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.692897 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.753152 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:33.753116 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13e305d78555894578d5848a397f831.slice/crio-8e385376f383b64adfd89a38d322765a41b74fdcc3b2f19165a358ee53c4123d WatchSource:0}: Error finding container 8e385376f383b64adfd89a38d322765a41b74fdcc3b2f19165a358ee53c4123d: Status 404 returned error can't find the container with id 8e385376f383b64adfd89a38d322765a41b74fdcc3b2f19165a358ee53c4123d Jul 19 07:42:33.760717 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.760694 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 07:42:33.764537 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:33.764513 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235d857ab46d455762737cb4a14294eb.slice/crio-6ba7431fc4eb10bd43f3f7ef34f12c0ed1aba0289522a11afdc221a65e676171 WatchSource:0}: Error finding container 6ba7431fc4eb10bd43f3f7ef34f12c0ed1aba0289522a11afdc221a65e676171: Status 404 returned error can't find the container with id 6ba7431fc4eb10bd43f3f7ef34f12c0ed1aba0289522a11afdc221a65e676171 Jul 19 07:42:33.794076 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:33.794049 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-126.ec2.internal\" not found" Jul 19 07:42:33.808467 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.808448 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:33.891112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.891071 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" Jul 19 07:42:33.901524 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.901500 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 07:42:33.903327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.903313 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" Jul 19 07:42:33.913488 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.913467 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 07:42:33.918931 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:33.918914 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:34.120912 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.120883 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:34.569945 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.569844 2574 apiserver.go:52] "Watching apiserver" Jul 19 07:42:34.575721 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.575690 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 19 07:42:34.577081 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.577048 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vkbl7","kube-system/konnectivity-agent-q6lgs","kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82","openshift-cluster-node-tuning-operator/tuned-zz5qh","openshift-dns/node-resolver-qnz7d","openshift-image-registry/node-ca-d59n6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal","openshift-multus/multus-6h2ps","openshift-multus/multus-additional-cni-plugins-qb24p","openshift-multus/network-metrics-daemon-4znm8","openshift-network-diagnostics/network-check-target-4xv94","openshift-network-operator/iptables-alerter-zm945","openshift-ovn-kubernetes/ovnkube-node-w854p"] Jul 19 07:42:34.579956 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.579926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.582379 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.582354 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7b449\"" Jul 19 07:42:34.582683 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.582663 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 19 07:42:34.582860 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.582703 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.582860 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.582704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.582860 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.582817 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.585278 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.585258 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 19 07:42:34.586327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.585624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkr9x\"" Jul 19 07:42:34.586327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.585834 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.586327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.585858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.586327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.585912 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.588400 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.588384 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 19 07:42:34.588495 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.588438 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 19 07:42:34.588641 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.588615 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.588745 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.588659 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.588860 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.588843 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 19 07:42:34.588930 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.588911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6bd7\"" Jul 19 07:42:34.589092 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.589077 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 19 07:42:34.590425 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.590406 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 19 07:42:34.599717 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.599664 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.602230 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.602198 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.602602 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.602580 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkqft\"" Jul 19 07:42:34.602842 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.602824 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 19 07:42:34.603106 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.603090 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.604003 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.603983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.604635 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.604605 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 07:42:34.605316 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-log-socket\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.605613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-socket-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.605613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-sys-fs\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.605613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f723a1d-e34e-4931-83ef-78d788f3c186-ovn-node-metrics-cert\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-ovnkube-script-lib\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-kubelet\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-ovnkube-config\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-slash\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-ovn\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-cni-netd\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-registration-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-run-ovn-kubernetes\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-cni-bin\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.605963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbh6\" (UniqueName: \"kubernetes.io/projected/8f723a1d-e34e-4931-83ef-78d788f3c186-kube-api-access-bkbh6\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606011 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-device-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-systemd-units\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-systemd\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-var-lib-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchnh\" (UniqueName: \"kubernetes.io/projected/0fd47b20-ad9d-437e-9630-f5b752a35a35-kube-api-access-pchnh\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-serviceca\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606309 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.606339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cdaf012e-983e-4fb0-abc4-fbed99260776-iptables-alerter-script\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdaf012e-983e-4fb0-abc4-fbed99260776-host-slash\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbk2\" (UniqueName: \"kubernetes.io/projected/cdaf012e-983e-4fb0-abc4-fbed99260776-kube-api-access-clbk2\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606474 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5qkv5\"" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-env-overrides\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-etc-selinux\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-host\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-run-netns\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-etc-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.606800 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.606709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-node-log\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.607481 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.607460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5jr\" (UniqueName: \"kubernetes.io/projected/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-kube-api-access-6v5jr\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.607725 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.607648 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.609597 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.609575 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.609943 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.609923 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.610080 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.610059 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bnq79\"" Jul 19 07:42:34.610529 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.610468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.613264 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.613243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.613365 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.613313 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:34.613701 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.613657 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 19 07:42:34.613937 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.613896 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 19 07:42:34.613937 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.613922 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 19 07:42:34.614147 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.614129 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j6g72\"" Jul 19 07:42:34.614206 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.614190 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 19 07:42:34.616227 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.615847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.618653 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.618636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.619161 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.619137 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m46r8\"" Jul 19 07:42:34.619265 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.619143 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 19 07:42:34.619433 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.619410 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 19 07:42:34.621148 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.621129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:34.621250 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.621216 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:34.621513 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.621493 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 19 07:42:34.621582 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.621517 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w5b7k\"" Jul 19 07:42:34.621582 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.621548 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 19 07:42:34.625052 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.625034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:34.625141 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.625098 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:34.625283 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.625262 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sfwbg" Jul 19 07:42:34.633439 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.633421 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sfwbg" Jul 19 07:42:34.691122 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.691091 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-socket-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-sys-fs\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-cnibin\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f723a1d-e34e-4931-83ef-78d788f3c186-ovn-node-metrics-cert\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-ovnkube-script-lib\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-kubelet\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-ovnkube-config\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-sys-fs\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708409 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708622 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-socket-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-cni-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.708981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-kubernetes\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-slash\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-registration-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-registration-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.711410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-run-ovn-kubernetes\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-run-ovn-kubernetes\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709281 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-slash\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbh6\" (UniqueName: \"kubernetes.io/projected/8f723a1d-e34e-4931-83ef-78d788f3c186-kube-api-access-bkbh6\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-device-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709455 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-ovnkube-script-lib\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4nb\" (UniqueName: \"kubernetes.io/projected/3578e380-7d65-4015-a9a0-acf3ee6b5456-kube-api-access-9p4nb\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-systemd\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-kubelet\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jtj\" (UniqueName: \"kubernetes.io/projected/e9cc735d-0114-4c55-839c-574f1c90d2bf-kube-api-access-56jtj\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-device-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.709699 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-run\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-var-lib-kubelet\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-ovnkube-config\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.712402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-hostroot\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdaf012e-983e-4fb0-abc4-fbed99260776-host-slash\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-env-overrides\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-systemd\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdaf012e-983e-4fb0-abc4-fbed99260776-host-slash\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-etc-selinux\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-etc-selinux\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-daemon-config\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-os-release\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-node-log\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036351e7-17ab-4846-8d06-e67caf447ba0-tmp-dir\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-host\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-node-log\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870abfa2-ba6e-4375-b66f-1a2a309118fb-agent-certs\") pod \"konnectivity-agent-q6lgs\" (UID: \"870abfa2-ba6e-4375-b66f-1a2a309118fb\") " pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-os-release\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-socket-dir-parent\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-cni-bin\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-log-socket\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f723a1d-e34e-4931-83ef-78d788f3c186-env-overrides\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-etc-kubernetes\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.710999 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-system-cni-dir\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-systemd\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-tmp\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fqm\" (UniqueName: \"kubernetes.io/projected/036351e7-17ab-4846-8d06-e67caf447ba0-kube-api-access-z2fqm\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-multus-certs\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fd47b20-ad9d-437e-9630-f5b752a35a35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-lib-modules\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-tuned\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-log-socket\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-ovn\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711381 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-ovn\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.713969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-cni-netd\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870abfa2-ba6e-4375-b66f-1a2a309118fb-konnectivity-ca\") pod \"konnectivity-agent-q6lgs\" (UID: \"870abfa2-ba6e-4375-b66f-1a2a309118fb\") " pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711483 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-cni-netd\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5x9\" (UniqueName: \"kubernetes.io/projected/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-kube-api-access-vk5x9\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bff89a7-ac8c-48fe-badc-c78dd9f68166-kubelet-config\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-cni-bin\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-conf-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-cni-bin\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-sys\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-kubelet\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-systemd-units\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-var-lib-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchnh\" (UniqueName: \"kubernetes.io/projected/0fd47b20-ad9d-437e-9630-f5b752a35a35-kube-api-access-pchnh\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmsp\" (UniqueName: \"kubernetes.io/projected/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-kube-api-access-xmmsp\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-systemd-units\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.714485 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-cnibin\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.711983 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-var-lib-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-netns\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-cni-multus\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712269 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-k8s-cni-cncf-io\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-serviceca\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cdaf012e-983e-4fb0-abc4-fbed99260776-iptables-alerter-script\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbk2\" (UniqueName: \"kubernetes.io/projected/cdaf012e-983e-4fb0-abc4-fbed99260776-kube-api-access-clbk2\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysconfig\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-serviceca\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysctl-conf\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-run-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-host\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-host\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-run-netns\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715112 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-host-run-netns\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.712996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cdaf012e-983e-4fb0-abc4-fbed99260776-iptables-alerter-script\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-etc-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-system-cni-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f723a1d-e34e-4931-83ef-78d788f3c186-etc-openvswitch\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3578e380-7d65-4015-a9a0-acf3ee6b5456-cni-binary-copy\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5jr\" (UniqueName: \"kubernetes.io/projected/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-kube-api-access-6v5jr\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/036351e7-17ab-4846-8d06-e67caf447ba0-hosts-file\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-modprobe-d\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysctl-d\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.715772 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.713815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bff89a7-ac8c-48fe-badc-c78dd9f68166-dbus\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.717110 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.717082 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f723a1d-e34e-4931-83ef-78d788f3c186-ovn-node-metrics-cert\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.720630 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.720606 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbh6\" (UniqueName: \"kubernetes.io/projected/8f723a1d-e34e-4931-83ef-78d788f3c186-kube-api-access-bkbh6\") pod \"ovnkube-node-w854p\" (UID: \"8f723a1d-e34e-4931-83ef-78d788f3c186\") " pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.724073 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.724047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchnh\" (UniqueName: \"kubernetes.io/projected/0fd47b20-ad9d-437e-9630-f5b752a35a35-kube-api-access-pchnh\") pod \"aws-ebs-csi-driver-node-nht82\" (UID: \"0fd47b20-ad9d-437e-9630-f5b752a35a35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.724235 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.724215 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5jr\" (UniqueName: \"kubernetes.io/projected/41e5e5d2-41f9-43b6-b303-ef986ddb5a26-kube-api-access-6v5jr\") pod \"node-ca-d59n6\" (UID: \"41e5e5d2-41f9-43b6-b303-ef986ddb5a26\") " pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.724604 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.724587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbk2\" (UniqueName: \"kubernetes.io/projected/cdaf012e-983e-4fb0-abc4-fbed99260776-kube-api-access-clbk2\") pod \"iptables-alerter-zm945\" (UID: \"cdaf012e-983e-4fb0-abc4-fbed99260776\") " pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.731910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.731828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" event={"ID":"235d857ab46d455762737cb4a14294eb","Type":"ContainerStarted","Data":"6ba7431fc4eb10bd43f3f7ef34f12c0ed1aba0289522a11afdc221a65e676171"} Jul 19 07:42:34.733219 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.733192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" event={"ID":"a13e305d78555894578d5848a397f831","Type":"ContainerStarted","Data":"8e385376f383b64adfd89a38d322765a41b74fdcc3b2f19165a358ee53c4123d"} Jul 19 07:42:34.787267 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.787232 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 07:42:34.814192 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bff89a7-ac8c-48fe-badc-c78dd9f68166-dbus\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-cnibin\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-cni-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-kubernetes\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4nb\" (UniqueName: \"kubernetes.io/projected/3578e380-7d65-4015-a9a0-acf3ee6b5456-kube-api-access-9p4nb\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-cnibin\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bff89a7-ac8c-48fe-badc-c78dd9f68166-dbus\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.814384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-kubernetes\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814366 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-cni-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jtj\" (UniqueName: \"kubernetes.io/projected/e9cc735d-0114-4c55-839c-574f1c90d2bf-kube-api-access-56jtj\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-run\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-var-lib-kubelet\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814651 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-run\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-hostroot\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-var-lib-kubelet\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814721 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-hostroot\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-daemon-config\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-os-release\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036351e7-17ab-4846-8d06-e67caf447ba0-tmp-dir\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-host\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870abfa2-ba6e-4375-b66f-1a2a309118fb-agent-certs\") pod \"konnectivity-agent-q6lgs\" (UID: \"870abfa2-ba6e-4375-b66f-1a2a309118fb\") " pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.814897 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:34.814894 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-os-release\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-socket-dir-parent\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-host\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.814905 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-os-release\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.814999 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:42:35.3149516 +0000 UTC m=+3.094614910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-socket-dir-parent\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-cni-bin\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-cni-bin\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-etc-kubernetes\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-etc-kubernetes\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-system-cni-dir\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-systemd\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-tmp\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fqm\" (UniqueName: \"kubernetes.io/projected/036351e7-17ab-4846-8d06-e67caf447ba0-kube-api-access-z2fqm\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-system-cni-dir\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036351e7-17ab-4846-8d06-e67caf447ba0-tmp-dir\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.815910 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-multus-certs\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815279 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-systemd\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-multus-certs\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-os-release\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-lib-modules\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-tuned\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-daemon-config\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870abfa2-ba6e-4375-b66f-1a2a309118fb-konnectivity-ca\") pod \"konnectivity-agent-q6lgs\" (UID: \"870abfa2-ba6e-4375-b66f-1a2a309118fb\") " pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5x9\" (UniqueName: \"kubernetes.io/projected/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-kube-api-access-vk5x9\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bff89a7-ac8c-48fe-badc-c78dd9f68166-kubelet-config\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-lib-modules\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-conf-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-sys\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bff89a7-ac8c-48fe-badc-c78dd9f68166-kubelet-config\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-kubelet\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.816768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-multus-conf-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmsp\" (UniqueName: \"kubernetes.io/projected/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-kube-api-access-xmmsp\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-cnibin\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-netns\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-cni-multus\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-k8s-cni-cncf-io\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-sys\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysconfig\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysctl-conf\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9cc735d-0114-4c55-839c-574f1c90d2bf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.815929 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-system-cni-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3578e380-7d65-4015-a9a0-acf3ee6b5456-cni-binary-copy\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.815966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-netns\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.816005 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:35.315980236 +0000 UTC m=+3.095643529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:34.818019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-kubelet\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/036351e7-17ab-4846-8d06-e67caf447ba0-hosts-file\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-modprobe-d\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysctl-d\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-cnibin\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysctl-d\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816323 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-var-lib-cni-multus\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-modprobe-d\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/036351e7-17ab-4846-8d06-e67caf447ba0-hosts-file\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-host-run-k8s-cni-cncf-io\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870abfa2-ba6e-4375-b66f-1a2a309118fb-konnectivity-ca\") pod \"konnectivity-agent-q6lgs\" (UID: \"870abfa2-ba6e-4375-b66f-1a2a309118fb\") " pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysctl-conf\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3578e380-7d65-4015-a9a0-acf3ee6b5456-cni-binary-copy\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-sysconfig\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3578e380-7d65-4015-a9a0-acf3ee6b5456-system-cni-dir\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.816873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9cc735d-0114-4c55-839c-574f1c90d2bf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.817820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-etc-tuned\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.818903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.818114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870abfa2-ba6e-4375-b66f-1a2a309118fb-agent-certs\") pod \"konnectivity-agent-q6lgs\" (UID: \"870abfa2-ba6e-4375-b66f-1a2a309118fb\") " pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.819699 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.818985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-tmp\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.821907 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.821850 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:34.821907 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.821875 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:34.821907 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.821909 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:34.822109 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:34.821982 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:35.321964366 +0000 UTC m=+3.101627677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:34.824652 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.824610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4nb\" (UniqueName: \"kubernetes.io/projected/3578e380-7d65-4015-a9a0-acf3ee6b5456-kube-api-access-9p4nb\") pod \"multus-6h2ps\" (UID: \"3578e380-7d65-4015-a9a0-acf3ee6b5456\") " pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.824935 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.824916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5x9\" (UniqueName: \"kubernetes.io/projected/34cb64ef-0f2e-47bc-91a9-abbaa26cafcf-kube-api-access-vk5x9\") pod \"tuned-zz5qh\" (UID: \"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf\") " pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.824935 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.824926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fqm\" (UniqueName: \"kubernetes.io/projected/036351e7-17ab-4846-8d06-e67caf447ba0-kube-api-access-z2fqm\") pod \"node-resolver-qnz7d\" (UID: \"036351e7-17ab-4846-8d06-e67caf447ba0\") " pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.825426 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.825407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmsp\" (UniqueName: \"kubernetes.io/projected/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-kube-api-access-xmmsp\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:34.825504 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.825449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jtj\" (UniqueName: \"kubernetes.io/projected/e9cc735d-0114-4c55-839c-574f1c90d2bf-kube-api-access-56jtj\") pod \"multus-additional-cni-plugins-qb24p\" (UID: \"e9cc735d-0114-4c55-839c-574f1c90d2bf\") " pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:34.903774 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.903728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d59n6" Jul 19 07:42:34.913858 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.913828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zm945" Jul 19 07:42:34.924825 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.924796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:42:34.933682 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.933659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" Jul 19 07:42:34.940807 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.940780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" Jul 19 07:42:34.947719 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.947697 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qnz7d" Jul 19 07:42:34.956559 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.956539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6h2ps" Jul 19 07:42:34.966623 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.966601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:34.972529 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:34.972512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qb24p" Jul 19 07:42:35.319227 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:35.319189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:35.319442 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:35.319293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:35.319442 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.319376 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:35.319442 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.319412 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:35.319577 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.319447 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:36.31942497 +0000 UTC m=+4.099088263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:35.319577 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.319472 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:42:36.319455854 +0000 UTC m=+4.099119170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:35.420824 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:35.420781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:35.421033 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.420953 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:35.421033 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.420984 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:35.421033 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.420996 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:35.421195 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.421066 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:36.421047522 +0000 UTC m=+4.200710815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:35.636528 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:35.636477 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 07:37:34 +0000 UTC" deadline="2028-01-29 03:11:54.835333049 +0000 UTC" Jul 19 07:42:35.636528 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:35.636507 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13411h29m19.198828842s" Jul 19 07:42:35.726344 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:35.726308 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:35.726546 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:35.726445 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:35.903980 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.903951 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd47b20_ad9d_437e_9630_f5b752a35a35.slice/crio-29ca6bd6c9952037839135a917a0ccec6d791b93cc337ed2210ac2b05cd4b030 WatchSource:0}: Error finding container 29ca6bd6c9952037839135a917a0ccec6d791b93cc337ed2210ac2b05cd4b030: Status 404 returned error can't find the container with id 29ca6bd6c9952037839135a917a0ccec6d791b93cc337ed2210ac2b05cd4b030 Jul 19 07:42:35.904756 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.904703 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3578e380_7d65_4015_a9a0_acf3ee6b5456.slice/crio-566ecd3c49471034b7c0a0db95ef890957afddd3289c576999b763741b612139 WatchSource:0}: Error finding container 566ecd3c49471034b7c0a0db95ef890957afddd3289c576999b763741b612139: Status 404 returned error can't find the container with id 566ecd3c49471034b7c0a0db95ef890957afddd3289c576999b763741b612139 Jul 19 07:42:35.908340 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.908314 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdaf012e_983e_4fb0_abc4_fbed99260776.slice/crio-7558426f2dfc2871e067ac9bd64d2cb15da728fbe40a3b73edecc1f801bd202f WatchSource:0}: Error finding container 7558426f2dfc2871e067ac9bd64d2cb15da728fbe40a3b73edecc1f801bd202f: Status 404 returned error can't find the container with id 7558426f2dfc2871e067ac9bd64d2cb15da728fbe40a3b73edecc1f801bd202f Jul 19 07:42:35.911022 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.910999 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e5e5d2_41f9_43b6_b303_ef986ddb5a26.slice/crio-b81194d97c5df7b3bf34073f2394203eaced62ab49e894605679e3dd9a76c290 WatchSource:0}: Error finding container b81194d97c5df7b3bf34073f2394203eaced62ab49e894605679e3dd9a76c290: Status 404 returned error can't find the container with id b81194d97c5df7b3bf34073f2394203eaced62ab49e894605679e3dd9a76c290 Jul 19 07:42:35.919067 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.919044 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870abfa2_ba6e_4375_b66f_1a2a309118fb.slice/crio-86022b63dee3f050e80b5f81bfa71dcb77d81c5c47ee601a680127ea68d90711 WatchSource:0}: Error finding container 86022b63dee3f050e80b5f81bfa71dcb77d81c5c47ee601a680127ea68d90711: Status 404 returned error can't find the container with id 86022b63dee3f050e80b5f81bfa71dcb77d81c5c47ee601a680127ea68d90711 Jul 19 07:42:35.920033 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.920007 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f723a1d_e34e_4931_83ef_78d788f3c186.slice/crio-ab410ff52f316cae78e7482080ea2c2f7e9a6bf0986d484aff54c08ff718413d WatchSource:0}: Error finding container ab410ff52f316cae78e7482080ea2c2f7e9a6bf0986d484aff54c08ff718413d: Status 404 returned error can't find the container with id ab410ff52f316cae78e7482080ea2c2f7e9a6bf0986d484aff54c08ff718413d Jul 19 07:42:35.926013 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:42:35.922395 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036351e7_17ab_4846_8d06_e67caf447ba0.slice/crio-147eb86d941c4af713575249be75b6b593aa7a106754dcdea2f5a4d85dcbf499 WatchSource:0}: Error finding container 147eb86d941c4af713575249be75b6b593aa7a106754dcdea2f5a4d85dcbf499: Status 404 returned error can't find the container with id 147eb86d941c4af713575249be75b6b593aa7a106754dcdea2f5a4d85dcbf499 Jul 19 07:42:36.329286 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.329141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:36.329286 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.329227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:36.329542 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.329314 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:36.329542 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.329344 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:36.329542 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.329392 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:38.329376069 +0000 UTC m=+6.109039367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:36.329542 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.329411 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:42:38.329402742 +0000 UTC m=+6.109066035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:36.430514 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.430469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:36.430731 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.430705 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:36.430820 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.430750 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:36.430820 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.430766 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:36.430943 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.430830 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:38.430810977 +0000 UTC m=+6.210474292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:36.636815 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.636740 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 07:37:34 +0000 UTC" deadline="2028-03-01 07:32:16.041167888 +0000 UTC" Jul 19 07:42:36.636815 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.636812 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14183h49m39.404360816s" Jul 19 07:42:36.729792 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.726518 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:36.729792 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.726649 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:36.729792 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.726765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:36.729792 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:36.726856 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:36.748320 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.748272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q6lgs" event={"ID":"870abfa2-ba6e-4375-b66f-1a2a309118fb","Type":"ContainerStarted","Data":"86022b63dee3f050e80b5f81bfa71dcb77d81c5c47ee601a680127ea68d90711"} Jul 19 07:42:36.774387 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.774229 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" event={"ID":"0fd47b20-ad9d-437e-9630-f5b752a35a35","Type":"ContainerStarted","Data":"29ca6bd6c9952037839135a917a0ccec6d791b93cc337ed2210ac2b05cd4b030"} Jul 19 07:42:36.786615 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.785588 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" event={"ID":"a13e305d78555894578d5848a397f831","Type":"ContainerStarted","Data":"6887cf489935f86bdf9e7167258aa28206cb1f3c5b5605d14930b4ca29a186dc"} Jul 19 07:42:36.793412 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.793361 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnz7d" event={"ID":"036351e7-17ab-4846-8d06-e67caf447ba0","Type":"ContainerStarted","Data":"147eb86d941c4af713575249be75b6b593aa7a106754dcdea2f5a4d85dcbf499"} Jul 19 07:42:36.802440 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.802403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"ab410ff52f316cae78e7482080ea2c2f7e9a6bf0986d484aff54c08ff718413d"} Jul 19 07:42:36.802936 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.802878 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-126.ec2.internal" podStartSLOduration=2.802858245 podStartE2EDuration="2.802858245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:36.802654376 +0000 UTC m=+4.582317695" watchObservedRunningTime="2026-07-19 07:42:36.802858245 +0000 UTC m=+4.582521563" Jul 19 07:42:36.807730 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.807699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" event={"ID":"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf","Type":"ContainerStarted","Data":"325ad274fd21d504af46292f60db130de2a2ce945556a32854ca9907fdc693d9"} Jul 19 07:42:36.812263 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.812059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerStarted","Data":"41dac4a99caf745108ad2fef8934f89544a85d4c54c9279a7a42d044abe7c6eb"} Jul 19 07:42:36.815054 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.815025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d59n6" event={"ID":"41e5e5d2-41f9-43b6-b303-ef986ddb5a26","Type":"ContainerStarted","Data":"b81194d97c5df7b3bf34073f2394203eaced62ab49e894605679e3dd9a76c290"} Jul 19 07:42:36.825367 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.825338 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zm945" event={"ID":"cdaf012e-983e-4fb0-abc4-fbed99260776","Type":"ContainerStarted","Data":"7558426f2dfc2871e067ac9bd64d2cb15da728fbe40a3b73edecc1f801bd202f"} Jul 19 07:42:36.833642 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.833581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6h2ps" event={"ID":"3578e380-7d65-4015-a9a0-acf3ee6b5456","Type":"ContainerStarted","Data":"566ecd3c49471034b7c0a0db95ef890957afddd3289c576999b763741b612139"} Jul 19 07:42:36.852099 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.851406 2574 generic.go:356] "Generic (PLEG): container finished" podID="235d857ab46d455762737cb4a14294eb" containerID="9207c3bed31180ba30cd93cc0ff2432089c40156f65e3a7844d8add549ea791d" exitCode=0 Jul 19 07:42:36.852099 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:36.851462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" event={"ID":"235d857ab46d455762737cb4a14294eb","Type":"ContainerDied","Data":"9207c3bed31180ba30cd93cc0ff2432089c40156f65e3a7844d8add549ea791d"} Jul 19 07:42:37.726086 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:37.726045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:37.726608 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:37.726213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:37.883775 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:37.880702 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" event={"ID":"235d857ab46d455762737cb4a14294eb","Type":"ContainerStarted","Data":"5469dfbbc0f3c69224725dc4fdb568719334b35b2a580d8bc1bd9d52b30b33dd"} Jul 19 07:42:38.350498 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:38.350457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:38.350748 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:38.350559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:38.350835 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.350812 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:38.350903 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.350896 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:42.350872298 +0000 UTC m=+10.130535613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:38.351395 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.350707 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:38.351395 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.351372 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:42:42.35135185 +0000 UTC m=+10.131015149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:38.451840 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:38.451795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:38.452109 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.452085 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:38.452213 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.452122 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:38.452213 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.452136 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:38.452213 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.452203 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:42.452181645 +0000 UTC m=+10.231844974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:38.726640 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:38.726554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:38.727277 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:38.727137 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:38.731021 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.730547 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:38.731021 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:38.730599 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:39.726213 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:39.726178 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:39.726400 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:39.726346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:40.728045 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:40.728012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:40.728468 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:40.728143 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:40.728545 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:40.728500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:40.728610 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:40.728588 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:41.726516 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:41.726484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:41.726820 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:41.726622 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:42.386411 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:42.386367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:42.386939 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:42.386468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:42.386939 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.386606 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:42.386939 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.386668 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:42:50.386649116 +0000 UTC m=+18.166312414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:42.387108 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.387007 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:42.387108 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.387046 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:50.387036266 +0000 UTC m=+18.166699557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:42.487078 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:42.487038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:42.487277 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.487264 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:42.487343 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.487281 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:42.487343 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.487291 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:42.487343 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.487337 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:50.487323638 +0000 UTC m=+18.266986933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:42.728506 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:42.728423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:42.728691 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.728538 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:42.728876 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:42.728859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:42.729707 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:42.728946 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:43.726059 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:43.726021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:43.726489 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:43.726156 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:44.727188 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:44.726800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:44.727188 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:44.726902 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:44.727712 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:44.727301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:44.727712 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:44.727406 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:45.726096 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:45.726060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:45.726272 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:45.726203 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:46.726455 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:46.726426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:46.726877 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:46.726426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:46.726877 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:46.726548 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:46.726877 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:46.726635 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:47.726356 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:47.726282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:47.726508 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:47.726416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:48.726306 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:48.726266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:48.726471 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:48.726322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:48.726471 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:48.726416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:48.726889 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:48.726538 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:49.725946 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:49.725908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:49.726205 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:49.726061 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:50.452496 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:50.452446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:50.453081 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:50.452534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:50.453081 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.452619 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:50.453081 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.452645 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:50.453081 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.452704 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.452670298 +0000 UTC m=+34.232333590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:42:50.453081 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.452724 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.452715165 +0000 UTC m=+34.232378457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:42:50.553024 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:50.552983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:50.553218 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.553195 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:42:50.553276 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.553230 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:42:50.553276 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.553246 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:50.553395 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.553309 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:06.553292535 +0000 UTC m=+34.332955831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:42:50.726615 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:50.726542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:50.726795 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.726666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:50.726795 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:50.726725 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:50.726917 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:50.726849 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:51.726265 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:51.726224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:51.726734 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:51.726379 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:52.726871 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:52.726838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:52.727364 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:52.726924 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:52.727364 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:52.726993 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:52.727364 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:52.727102 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:53.726775 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:53.726730 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:53.726971 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:53.726874 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:54.726075 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:54.726035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:54.726359 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:54.726185 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:54.726359 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:54.726240 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:54.726359 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:54.726349 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:55.727018 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.726746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:55.727643 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:55.727105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:55.920273 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.920241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6h2ps" event={"ID":"3578e380-7d65-4015-a9a0-acf3ee6b5456","Type":"ContainerStarted","Data":"78b73616a22e4d389b2d16e1dcc282ae164c489737ceab59d3b8b2f9fae2b3eb"} Jul 19 07:42:55.921805 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.921712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q6lgs" event={"ID":"870abfa2-ba6e-4375-b66f-1a2a309118fb","Type":"ContainerStarted","Data":"7278dc54dd96435b147ceca72840ac3e9e4dd45db1e02bda22db986865ddbc75"} Jul 19 07:42:55.923153 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.923132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" event={"ID":"0fd47b20-ad9d-437e-9630-f5b752a35a35","Type":"ContainerStarted","Data":"619ef1689c8d340d30ce63a2c038bed4028b321fb2cf3d54057e026471eac76f"} Jul 19 07:42:55.924470 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.924451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qnz7d" event={"ID":"036351e7-17ab-4846-8d06-e67caf447ba0","Type":"ContainerStarted","Data":"68b76aed9112a6fae0f728819c9419161239af0ddbbfdc9f738ff767b9da87b5"} Jul 19 07:42:55.926286 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.926267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"404ef45389243af3805c208698c2ae197a734f680bf0a8756a6c78592ef47043"} Jul 19 07:42:55.926387 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.926322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"acf4b744fe372eb0b4eba13733d42ecabdcb0a9d67f60e5099051440bfe2f387"} Jul 19 07:42:55.926387 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.926357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"57ce746521bddf3dbb26ec184863607d5237839524b6dfc5eebe65b8afbc3ab0"} Jul 19 07:42:55.927627 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.927611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" event={"ID":"34cb64ef-0f2e-47bc-91a9-abbaa26cafcf","Type":"ContainerStarted","Data":"c3ff03d92247164cbec38d37f036771043473bfe60c4a18f486551e2ffd259b9"} Jul 19 07:42:55.928877 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.928856 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9cc735d-0114-4c55-839c-574f1c90d2bf" containerID="64196c96e08217cccc11a0baba53b48a46751f5bdc1db887e5b1aba0b7c83fa3" exitCode=0 Jul 19 07:42:55.928947 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.928914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerDied","Data":"64196c96e08217cccc11a0baba53b48a46751f5bdc1db887e5b1aba0b7c83fa3"} Jul 19 07:42:55.930231 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.930212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d59n6" event={"ID":"41e5e5d2-41f9-43b6-b303-ef986ddb5a26","Type":"ContainerStarted","Data":"29fa67ce41eec41f99b24bfbb1d165585c3c60fb87d88a98e403a47414a5236e"} Jul 19 07:42:55.935820 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.935784 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-126.ec2.internal" podStartSLOduration=21.935772793 podStartE2EDuration="21.935772793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:37.898155763 +0000 UTC m=+5.677819081" watchObservedRunningTime="2026-07-19 07:42:55.935772793 +0000 UTC m=+23.715436106" Jul 19 07:42:55.935909 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.935849 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6h2ps" podStartSLOduration=3.353272048 podStartE2EDuration="22.935845299s" totalImagesPullingTime="19.582573251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:55.935246765 +0000 UTC m=+23.714910079" watchObservedRunningTime="2026-07-19 07:42:55.935845299 +0000 UTC m=+23.715508612" Jul 19 07:42:55.948073 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.948030 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d59n6" podStartSLOduration=3.431796942 podStartE2EDuration="22.948015302s" totalImagesPullingTime="19.51621836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:55.9479621 +0000 UTC m=+23.727625413" watchObservedRunningTime="2026-07-19 07:42:55.948015302 +0000 UTC m=+23.727678617" Jul 19 07:42:55.963456 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.963414 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zz5qh" podStartSLOduration=3.453524076 podStartE2EDuration="22.963400491s" totalImagesPullingTime="19.509876415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:55.962820559 +0000 UTC m=+23.742483873" watchObservedRunningTime="2026-07-19 07:42:55.963400491 +0000 UTC m=+23.743063804" Jul 19 07:42:55.979545 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.979493 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qnz7d" podStartSLOduration=3.4782440279999998 podStartE2EDuration="22.979479597s" totalImagesPullingTime="19.501235569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:55.97922605 +0000 UTC m=+23.758889364" watchObservedRunningTime="2026-07-19 07:42:55.979479597 +0000 UTC m=+23.759142912" Jul 19 07:42:55.995183 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:55.995144 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q6lgs" podStartSLOduration=3.4878806620000002 podStartE2EDuration="22.995129277s" totalImagesPullingTime="19.507248615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:55.994729049 +0000 UTC m=+23.774392380" watchObservedRunningTime="2026-07-19 07:42:55.995129277 +0000 UTC m=+23.774792590" Jul 19 07:42:56.725981 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.725921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:56.726100 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:56.726036 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:56.726143 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.726098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:56.726226 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:56.726203 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:56.881192 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.881159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:56.881783 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.881764 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:56.935327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.935301 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"b1df293abbdfe36a40448c4ac04c4eb0eaad8d3026798013b5d45f3eb8b35ca3"} Jul 19 07:42:56.935462 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.935384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"ceefaa810e2375ab47199ca2abdcfab2ce90aeaa85c583a9aa68078cfff0bf9a"} Jul 19 07:42:56.935462 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.935435 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"bc6ed566a64c6f0d7e4f45c2b7fa84029701b733488cbb7079ff2a9d475cdb9d"} Jul 19 07:42:56.936668 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.936645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zm945" event={"ID":"cdaf012e-983e-4fb0-abc4-fbed99260776","Type":"ContainerStarted","Data":"45c5dca58e34e09b25da1408c80ba0870a626a0fa75cd6aa37fb422e35acd1b5"} Jul 19 07:42:56.942147 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.942122 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:56.942267 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.942190 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q6lgs" Jul 19 07:42:56.963940 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:56.963887 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zm945" podStartSLOduration=4.442313043 podStartE2EDuration="23.96386674s" totalImagesPullingTime="19.521553697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:56.95679572 +0000 UTC m=+24.736459034" watchObservedRunningTime="2026-07-19 07:42:56.96386674 +0000 UTC m=+24.743530055" Jul 19 07:42:57.061645 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:57.061610 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 19 07:42:57.678267 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:57.678149 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T07:42:57.061640162Z","UUID":"4540a422-1ca9-43f9-a6f3-1c6e46137387","Handler":null,"Name":"","Endpoint":""} Jul 19 07:42:57.686849 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:57.686822 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 19 07:42:57.687021 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:57.686863 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 19 07:42:57.725977 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:57.725946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:57.726151 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:57.726095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:42:57.941428 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:57.941332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" event={"ID":"0fd47b20-ad9d-437e-9630-f5b752a35a35","Type":"ContainerStarted","Data":"405d936be3a3d1e65d07b42ebf2b3336e7f336a8deb17230fe3c8f2e7c867675"} Jul 19 07:42:58.725881 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:58.725831 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:42:58.725881 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:58.725874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:42:58.726088 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:58.725981 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:42:58.726126 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:58.726096 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:42:58.948089 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:58.948058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"f2b10b6391c624db1f253bf45085d238e0f7043d290fa053cb9f16029d198c30"} Jul 19 07:42:58.949997 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:58.949966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" event={"ID":"0fd47b20-ad9d-437e-9630-f5b752a35a35","Type":"ContainerStarted","Data":"cab3e4ddb41f65a03ff2ae2afd6616f05f0c2030d31fabdb1e681c2f74c47935"} Jul 19 07:42:58.966192 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:58.966153 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nht82" podStartSLOduration=3.796725998 podStartE2EDuration="25.966140719s" totalImagesPullingTime="22.169414721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:58.965973451 +0000 UTC m=+26.745636764" watchObservedRunningTime="2026-07-19 07:42:58.966140719 +0000 UTC m=+26.745804030" Jul 19 07:42:59.726005 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:42:59.725829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:42:59.726169 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:42:59.726081 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:00.726461 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:00.726426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:00.726965 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:00.726477 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:00.726965 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:00.726556 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:00.726965 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:00.726701 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:01.726532 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.726368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:01.727136 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:01.726619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:01.964204 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.964116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" event={"ID":"8f723a1d-e34e-4931-83ef-78d788f3c186","Type":"ContainerStarted","Data":"1f87b66b7cab3a57a0998c0e9ec5cb0f155f02c1edbf16fbfbbba931f3cb18c8"} Jul 19 07:43:01.964379 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.964363 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:43:01.965949 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.965920 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9cc735d-0114-4c55-839c-574f1c90d2bf" containerID="30d72655e1b648c488343cd9d723809e8eb7c762e5786437e9daf52aa9e61a24" exitCode=0 Jul 19 07:43:01.966051 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.965976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerDied","Data":"30d72655e1b648c488343cd9d723809e8eb7c762e5786437e9daf52aa9e61a24"} Jul 19 07:43:01.980751 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.980723 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:43:01.987929 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:01.987885 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" podStartSLOduration=9.411926474 podStartE2EDuration="28.987870015s" totalImagesPullingTime="19.575943541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:01.986257383 +0000 UTC m=+29.765920696" watchObservedRunningTime="2026-07-19 07:43:01.987870015 +0000 UTC m=+29.767533330" Jul 19 07:43:02.726947 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:02.726904 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:02.727457 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:02.727032 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:02.727457 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:02.727420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:02.727544 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:02.727495 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:02.968025 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:02.967982 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:43:02.968130 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:02.968061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:43:02.984000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:02.983926 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:43:03.037703 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.037657 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnz7d_036351e7-17ab-4846-8d06-e67caf447ba0/dns-node-resolver/0.log" Jul 19 07:43:03.726505 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.726450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:03.726732 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:03.726628 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:03.793323 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.793289 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xv94"] Jul 19 07:43:03.793855 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.793404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:03.793855 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:03.793492 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:03.794067 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.794044 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vkbl7"] Jul 19 07:43:03.794154 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.794141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:03.794257 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:03.794239 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:03.795482 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.795462 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4znm8"] Jul 19 07:43:03.803838 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.803819 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d59n6_41e5e5d2-41f9-43b6-b303-ef986ddb5a26/node-ca/0.log" Jul 19 07:43:03.971656 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.971619 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9cc735d-0114-4c55-839c-574f1c90d2bf" containerID="bb12a32db508c20fa04da52d4103b9514295bbda5a244e671adce547c9e662e6" exitCode=0 Jul 19 07:43:03.971847 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.971714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerDied","Data":"bb12a32db508c20fa04da52d4103b9514295bbda5a244e671adce547c9e662e6"} Jul 19 07:43:03.971847 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:03.971758 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:03.971998 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:03.971965 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:05.726713 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:05.726651 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:05.727088 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:05.726659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:05.727088 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:05.726780 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:05.727088 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:05.726803 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:05.727088 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:05.726875 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:05.727088 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:05.726945 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:05.977823 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:05.977790 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9cc735d-0114-4c55-839c-574f1c90d2bf" containerID="0371f1ca60597624465a5a18146bd2a0febb7fb8104dc1454021d074d8ff602a" exitCode=0 Jul 19 07:43:05.977976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:05.977846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerDied","Data":"0371f1ca60597624465a5a18146bd2a0febb7fb8104dc1454021d074d8ff602a"} Jul 19 07:43:06.472201 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:06.472167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:06.472387 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:06.472228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:06.472387 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.472339 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:43:06.472387 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.472383 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 07:43:06.472560 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.472420 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs podName:f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c nodeName:}" failed. No retries permitted until 2026-07-19 07:43:38.472397961 +0000 UTC m=+66.252061255 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs") pod "network-metrics-daemon-4znm8" (UID: "f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 07:43:06.472560 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.472458 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret podName:8bff89a7-ac8c-48fe-badc-c78dd9f68166 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:38.472440333 +0000 UTC m=+66.252103634 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret") pod "global-pull-secret-syncer-vkbl7" (UID: "8bff89a7-ac8c-48fe-badc-c78dd9f68166") : object "kube-system"/"original-pull-secret" not registered Jul 19 07:43:06.573143 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:06.573109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:06.573302 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.573272 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 07:43:06.573302 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.573287 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 07:43:06.573302 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.573295 2574 projected.go:196] Error preparing data for projected volume kube-api-access-qn8nm for pod openshift-network-diagnostics/network-check-target-4xv94: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:43:06.573426 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:06.573349 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm podName:2075b26b-ce71-46f3-a8a4-15ac53430240 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:38.573333872 +0000 UTC m=+66.352997164 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qn8nm" (UniqueName: "kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm") pod "network-check-target-4xv94" (UID: "2075b26b-ce71-46f3-a8a4-15ac53430240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 07:43:07.726558 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:07.726523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:07.726558 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:07.726536 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:07.727109 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:07.726535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:07.727109 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:07.726651 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:07.727109 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:07.726764 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:07.727109 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:07.726876 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:09.726279 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:09.726225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:09.726911 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:09.726293 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:09.726911 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:09.726241 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:09.726911 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:09.726373 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:09.726911 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:09.726510 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:09.726911 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:09.726609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:10.993988 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:10.993950 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9cc735d-0114-4c55-839c-574f1c90d2bf" containerID="67d2c8ccd9b425784c4996a53a7d61f68db0f7908cf1400501b1829e396ad386" exitCode=0 Jul 19 07:43:10.994372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:10.994006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerDied","Data":"67d2c8ccd9b425784c4996a53a7d61f68db0f7908cf1400501b1829e396ad386"} Jul 19 07:43:11.726870 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:11.726838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:11.727033 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:11.726936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:11.727033 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:11.726954 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:11.727033 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:11.726965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:11.727160 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:11.727053 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:11.727160 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:11.727135 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:11.999789 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:11.999694 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9cc735d-0114-4c55-839c-574f1c90d2bf" containerID="c0cb302dfb7720c75fb4f48b3653ed523cec9ff43bd97c7ded4ec9c43fd68743" exitCode=0 Jul 19 07:43:11.999789 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:11.999742 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerDied","Data":"c0cb302dfb7720c75fb4f48b3653ed523cec9ff43bd97c7ded4ec9c43fd68743"} Jul 19 07:43:13.004339 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:13.004306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qb24p" event={"ID":"e9cc735d-0114-4c55-839c-574f1c90d2bf","Type":"ContainerStarted","Data":"398656f926b7e2ff6b4ef8b515ffa31f1a0b18e3f1e034b467c3d4e7306297c7"} Jul 19 07:43:13.026343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:13.026294 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qb24p" podStartSLOduration=8.585802656 podStartE2EDuration="40.026278009s" totalImagesPullingTime="31.440475353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:13.024793316 +0000 UTC m=+40.804456645" watchObservedRunningTime="2026-07-19 07:43:13.026278009 +0000 UTC m=+40.805941323" Jul 19 07:43:13.726345 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:13.726305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:13.726345 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:13.726348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:13.726564 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:13.726402 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:13.726564 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:13.726408 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:13.726564 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:13.726466 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:13.726564 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:13.726540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:15.726398 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:15.726364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:15.726398 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:15.726398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:15.726850 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:15.726364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:15.726850 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:15.726489 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:15.726850 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:15.726532 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:15.726850 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:15.726611 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:17.725956 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:17.725924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:17.726349 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:17.726026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:17.726349 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:17.726035 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:17.726349 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:17.726052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:17.726349 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:17.726119 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:17.726349 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:17.726176 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:19.726260 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:19.726228 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:19.726811 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:19.726228 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:19.726811 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:19.726346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:19.726811 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:19.726228 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:19.726811 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:19.726430 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:19.726811 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:19.726512 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:21.726031 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:21.725992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:21.726443 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:21.726091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:21.726443 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:21.726098 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:21.726443 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:21.726120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:21.726443 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:21.726198 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:21.726443 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:21.726263 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:23.726365 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:23.726333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:23.726791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:23.726440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:23.726791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:23.726528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:23.726791 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:23.726546 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xv94" podUID="2075b26b-ce71-46f3-a8a4-15ac53430240" Jul 19 07:43:23.726791 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:23.726595 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vkbl7" podUID="8bff89a7-ac8c-48fe-badc-c78dd9f68166" Jul 19 07:43:23.726791 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:23.726716 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4znm8" podUID="f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c" Jul 19 07:43:24.092917 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.092892 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-126.ec2.internal" event="NodeReady" Jul 19 07:43:24.093064 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.093013 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 19 07:43:24.126745 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.126710 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-845bbc4585-824dm"] Jul 19 07:43:24.150590 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.150564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845bbc4585-824dm"] Jul 19 07:43:24.150590 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.150600 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j6frs"] Jul 19 07:43:24.150790 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.150710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.153151 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.153127 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 19 07:43:24.153307 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.153240 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-thnlm\"" Jul 19 07:43:24.153479 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.153463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 19 07:43:24.153479 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.153477 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 19 07:43:24.159567 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.159546 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 19 07:43:24.166547 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.166524 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7tll8"] Jul 19 07:43:24.166693 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.166611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.169031 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.169012 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 19 07:43:24.169127 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.169044 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 19 07:43:24.169127 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.169013 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 19 07:43:24.169127 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.169013 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qmw7p\"" Jul 19 07:43:24.188646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.188619 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6frs"] Jul 19 07:43:24.188646 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.188648 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7tll8"] Jul 19 07:43:24.188812 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.188745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.190880 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.190861 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 19 07:43:24.190880 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.190861 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcblr\"" Jul 19 07:43:24.191052 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.190900 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 19 07:43:24.192206 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-bound-sa-token\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192294 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jm5g\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-kube-api-access-6jm5g\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192294 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4539355-46df-450c-bbe7-38024d9068a1-cert\") pod \"ingress-canary-j6frs\" (UID: \"c4539355-46df-450c-bbe7-38024d9068a1\") " pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.192394 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-trusted-ca\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192394 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-installation-pull-secrets\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192486 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-image-registry-private-configuration\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192486 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-ca-trust-extracted\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192583 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-registry-certificates\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.192583 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn926\" (UniqueName: \"kubernetes.io/projected/c4539355-46df-450c-bbe7-38024d9068a1-kube-api-access-kn926\") pod \"ingress-canary-j6frs\" (UID: \"c4539355-46df-450c-bbe7-38024d9068a1\") " pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.192583 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.192566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-registry-tls\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.242202 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.242176 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d75m9"] Jul 19 07:43:24.266207 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.266182 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d75m9"] Jul 19 07:43:24.266344 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.266302 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.268864 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.268846 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhks2\"" Jul 19 07:43:24.268975 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.268885 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 19 07:43:24.269152 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.269137 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 19 07:43:24.269217 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.269194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 19 07:43:24.269347 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.269291 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 19 07:43:24.293510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn926\" (UniqueName: \"kubernetes.io/projected/c4539355-46df-450c-bbe7-38024d9068a1-kube-api-access-kn926\") pod \"ingress-canary-j6frs\" (UID: \"c4539355-46df-450c-bbe7-38024d9068a1\") " pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.293619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-registry-tls\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.293619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4ff-66ba-413d-94e7-1a2988168d63-tmp-dir\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.293619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e52bb4ff-66ba-413d-94e7-1a2988168d63-metrics-tls\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.293784 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trds5\" (UniqueName: \"kubernetes.io/projected/e52bb4ff-66ba-413d-94e7-1a2988168d63-kube-api-access-trds5\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.293784 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02dfe2e2-f207-4040-9857-1b45b5267e07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.293784 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-bound-sa-token\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.293784 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jm5g\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-kube-api-access-6jm5g\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.293969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/02dfe2e2-f207-4040-9857-1b45b5267e07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.293969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4539355-46df-450c-bbe7-38024d9068a1-cert\") pod \"ingress-canary-j6frs\" (UID: \"c4539355-46df-450c-bbe7-38024d9068a1\") " pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.293969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzlxr\" (UniqueName: \"kubernetes.io/projected/02dfe2e2-f207-4040-9857-1b45b5267e07-kube-api-access-vzlxr\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.293969 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.293963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/02dfe2e2-f207-4040-9857-1b45b5267e07-crio-socket\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.294159 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-trusted-ca\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.294159 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-installation-pull-secrets\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.294159 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e52bb4ff-66ba-413d-94e7-1a2988168d63-config-volume\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.294159 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-image-registry-private-configuration\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.294343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-ca-trust-extracted\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.294343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/02dfe2e2-f207-4040-9857-1b45b5267e07-data-volume\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.294343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-registry-certificates\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.294569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.294547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-ca-trust-extracted\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.295100 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.295078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-trusted-ca\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.295263 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.295084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-registry-certificates\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.298227 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.298200 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4539355-46df-450c-bbe7-38024d9068a1-cert\") pod \"ingress-canary-j6frs\" (UID: \"c4539355-46df-450c-bbe7-38024d9068a1\") " pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.298227 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.298207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-installation-pull-secrets\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.298383 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.298250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-image-registry-private-configuration\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.298900 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.298875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-registry-tls\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.300975 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.300946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn926\" (UniqueName: \"kubernetes.io/projected/c4539355-46df-450c-bbe7-38024d9068a1-kube-api-access-kn926\") pod \"ingress-canary-j6frs\" (UID: \"c4539355-46df-450c-bbe7-38024d9068a1\") " pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.301092 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.301072 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jm5g\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-kube-api-access-6jm5g\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.301334 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.301319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8-bound-sa-token\") pod \"image-registry-845bbc4585-824dm\" (UID: \"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8\") " pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.395663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/02dfe2e2-f207-4040-9857-1b45b5267e07-data-volume\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.395663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4ff-66ba-413d-94e7-1a2988168d63-tmp-dir\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.395663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e52bb4ff-66ba-413d-94e7-1a2988168d63-metrics-tls\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.395663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trds5\" (UniqueName: \"kubernetes.io/projected/e52bb4ff-66ba-413d-94e7-1a2988168d63-kube-api-access-trds5\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.396015 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02dfe2e2-f207-4040-9857-1b45b5267e07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.396015 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/02dfe2e2-f207-4040-9857-1b45b5267e07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.396015 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzlxr\" (UniqueName: \"kubernetes.io/projected/02dfe2e2-f207-4040-9857-1b45b5267e07-kube-api-access-vzlxr\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.396015 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/02dfe2e2-f207-4040-9857-1b45b5267e07-crio-socket\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.396015 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.395804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e52bb4ff-66ba-413d-94e7-1a2988168d63-config-volume\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.396217 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.396035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/02dfe2e2-f207-4040-9857-1b45b5267e07-data-volume\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.396217 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.396050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52bb4ff-66ba-413d-94e7-1a2988168d63-tmp-dir\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.396217 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.396116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/02dfe2e2-f207-4040-9857-1b45b5267e07-crio-socket\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.396397 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.396375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e52bb4ff-66ba-413d-94e7-1a2988168d63-config-volume\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.396444 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.396400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/02dfe2e2-f207-4040-9857-1b45b5267e07-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.398210 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.398193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e52bb4ff-66ba-413d-94e7-1a2988168d63-metrics-tls\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.398297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.398277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02dfe2e2-f207-4040-9857-1b45b5267e07-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.403276 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.403253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trds5\" (UniqueName: \"kubernetes.io/projected/e52bb4ff-66ba-413d-94e7-1a2988168d63-kube-api-access-trds5\") pod \"dns-default-7tll8\" (UID: \"e52bb4ff-66ba-413d-94e7-1a2988168d63\") " pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.403436 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.403414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzlxr\" (UniqueName: \"kubernetes.io/projected/02dfe2e2-f207-4040-9857-1b45b5267e07-kube-api-access-vzlxr\") pod \"insights-runtime-extractor-d75m9\" (UID: \"02dfe2e2-f207-4040-9857-1b45b5267e07\") " pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.461378 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.461348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:24.475467 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.475441 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6frs" Jul 19 07:43:24.496656 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.496633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:24.575257 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.575223 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d75m9" Jul 19 07:43:24.692917 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.692880 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845bbc4585-824dm"] Jul 19 07:43:24.695024 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.695000 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6frs"] Jul 19 07:43:24.698433 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:24.698387 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd124b76_fc39_4a0b_9ada_f6f9aa15e8c8.slice/crio-3a70811474bf6c412abbad70050e8e8ab2aacb454e8ade5c7cbf284a420c2413 WatchSource:0}: Error finding container 3a70811474bf6c412abbad70050e8e8ab2aacb454e8ade5c7cbf284a420c2413: Status 404 returned error can't find the container with id 3a70811474bf6c412abbad70050e8e8ab2aacb454e8ade5c7cbf284a420c2413 Jul 19 07:43:24.700187 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:24.700128 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4539355_46df_450c_bbe7_38024d9068a1.slice/crio-ad1d1bcc0497750d4097cc1a2264814c6d811e8f6500478ad32968e7dde63050 WatchSource:0}: Error finding container ad1d1bcc0497750d4097cc1a2264814c6d811e8f6500478ad32968e7dde63050: Status 404 returned error can't find the container with id ad1d1bcc0497750d4097cc1a2264814c6d811e8f6500478ad32968e7dde63050 Jul 19 07:43:24.713645 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.713580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7tll8"] Jul 19 07:43:24.714632 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:24.714445 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52bb4ff_66ba_413d_94e7_1a2988168d63.slice/crio-77c26636955b086edfaf02dd02d979646cf965987feccf529a002346667aa03e WatchSource:0}: Error finding container 77c26636955b086edfaf02dd02d979646cf965987feccf529a002346667aa03e: Status 404 returned error can't find the container with id 77c26636955b086edfaf02dd02d979646cf965987feccf529a002346667aa03e Jul 19 07:43:24.748019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:24.747990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d75m9"] Jul 19 07:43:24.750463 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:24.750438 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dfe2e2_f207_4040_9857_1b45b5267e07.slice/crio-c16539fc0cf8cb5666fb83fa3e63eab0134237dc3e842aa80f4cbe0af9e1eb1c WatchSource:0}: Error finding container c16539fc0cf8cb5666fb83fa3e63eab0134237dc3e842aa80f4cbe0af9e1eb1c: Status 404 returned error can't find the container with id c16539fc0cf8cb5666fb83fa3e63eab0134237dc3e842aa80f4cbe0af9e1eb1c Jul 19 07:43:25.031946 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.031858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845bbc4585-824dm" event={"ID":"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8","Type":"ContainerStarted","Data":"481e8e423dcd6696167a33291116f75dc636321b862796919cb5687e5a46be65"} Jul 19 07:43:25.031946 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.031906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845bbc4585-824dm" event={"ID":"bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8","Type":"ContainerStarted","Data":"3a70811474bf6c412abbad70050e8e8ab2aacb454e8ade5c7cbf284a420c2413"} Jul 19 07:43:25.032154 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.031978 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:25.033175 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.033153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6frs" event={"ID":"c4539355-46df-450c-bbe7-38024d9068a1","Type":"ContainerStarted","Data":"ad1d1bcc0497750d4097cc1a2264814c6d811e8f6500478ad32968e7dde63050"} Jul 19 07:43:25.034228 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.034201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d75m9" event={"ID":"02dfe2e2-f207-4040-9857-1b45b5267e07","Type":"ContainerStarted","Data":"c16539fc0cf8cb5666fb83fa3e63eab0134237dc3e842aa80f4cbe0af9e1eb1c"} Jul 19 07:43:25.035377 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.035359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tll8" event={"ID":"e52bb4ff-66ba-413d-94e7-1a2988168d63","Type":"ContainerStarted","Data":"77c26636955b086edfaf02dd02d979646cf965987feccf529a002346667aa03e"} Jul 19 07:43:25.049250 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.049192 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-845bbc4585-824dm" podStartSLOduration=2.049173253 podStartE2EDuration="2.049173253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:25.048650554 +0000 UTC m=+52.828313868" watchObservedRunningTime="2026-07-19 07:43:25.049173253 +0000 UTC m=+52.828836561" Jul 19 07:43:25.726402 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.726365 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:25.726887 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.726866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:25.727300 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.727285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:25.729315 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.729275 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 07:43:25.730280 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.730258 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 07:43:25.730721 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.730703 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptfjq\"" Jul 19 07:43:25.730931 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.730913 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 07:43:25.731126 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.731109 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 07:43:25.731360 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:25.731343 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrpxw\"" Jul 19 07:43:31.058291 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.057859 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6frs" event={"ID":"c4539355-46df-450c-bbe7-38024d9068a1","Type":"ContainerStarted","Data":"5736821c56e57545290d913ab6e1a1ce3931f600a598d3ee997c5c4a23483c37"} Jul 19 07:43:31.060410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.060373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d75m9" event={"ID":"02dfe2e2-f207-4040-9857-1b45b5267e07","Type":"ContainerStarted","Data":"06179bb373e9d7921971b7cf3313bb4a7ef95b9ef9d800800f46d87416bb130b"} Jul 19 07:43:31.062647 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.062617 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tll8" event={"ID":"e52bb4ff-66ba-413d-94e7-1a2988168d63","Type":"ContainerStarted","Data":"dd8878b719d8b93ffadde0c9c964acc81aed67d3b860e5d90b4f76c388455be0"} Jul 19 07:43:31.062810 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.062703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tll8" event={"ID":"e52bb4ff-66ba-413d-94e7-1a2988168d63","Type":"ContainerStarted","Data":"f085dfedc6bfd8bf11b64e4f26e49ded8eeae51777d5a31d7bd0644e920f5feb"} Jul 19 07:43:31.062953 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.062930 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:31.074724 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.074628 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j6frs" podStartSLOduration=1.311744929 podStartE2EDuration="7.074608952s" totalImagesPullingTime="5.762864023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:31.073775035 +0000 UTC m=+58.853438352" watchObservedRunningTime="2026-07-19 07:43:31.074608952 +0000 UTC m=+58.854272267" Jul 19 07:43:31.095053 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:31.094301 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7tll8" podStartSLOduration=1.345899945 podStartE2EDuration="7.094278334s" totalImagesPullingTime="5.748378389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:31.093899308 +0000 UTC m=+58.873562624" watchObservedRunningTime="2026-07-19 07:43:31.094278334 +0000 UTC m=+58.873941648" Jul 19 07:43:33.989360 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:33.989327 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w854p" Jul 19 07:43:35.077574 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:35.077533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d75m9" event={"ID":"02dfe2e2-f207-4040-9857-1b45b5267e07","Type":"ContainerStarted","Data":"e1d8615c521b29b73049d9dadce376dfe14b6b99812a43ba754dbc5f9bd2cd3b"} Jul 19 07:43:38.088851 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.088817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d75m9" event={"ID":"02dfe2e2-f207-4040-9857-1b45b5267e07","Type":"ContainerStarted","Data":"37951bc25c4f064341aaeaec4a0000698b7fe440af59bc147385577ca91e42eb"} Jul 19 07:43:38.105110 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.105065 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d75m9" podStartSLOduration=0.992013776 podStartE2EDuration="14.105049361s" totalImagesPullingTime="13.113035585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:43:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:38.103953242 +0000 UTC m=+65.883616559" watchObservedRunningTime="2026-07-19 07:43:38.105049361 +0000 UTC m=+65.884712674" Jul 19 07:43:38.512285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.512246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:38.512463 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.512294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:38.514793 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.514773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 07:43:38.514840 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.514773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 07:43:38.524946 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.524919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bff89a7-ac8c-48fe-badc-c78dd9f68166-original-pull-secret\") pod \"global-pull-secret-syncer-vkbl7\" (UID: \"8bff89a7-ac8c-48fe-badc-c78dd9f68166\") " pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:38.525055 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.524966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c-metrics-certs\") pod \"network-metrics-daemon-4znm8\" (UID: \"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c\") " pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:38.613690 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.613634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:38.616823 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.616802 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 07:43:38.626841 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.626822 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 07:43:38.638048 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.638011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8nm\" (UniqueName: \"kubernetes.io/projected/2075b26b-ce71-46f3-a8a4-15ac53430240-kube-api-access-qn8nm\") pod \"network-check-target-4xv94\" (UID: \"2075b26b-ce71-46f3-a8a4-15ac53430240\") " pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:38.648475 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.648450 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptfjq\"" Jul 19 07:43:38.656354 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.656336 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:38.656437 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.656342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vkbl7" Jul 19 07:43:38.666036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.666012 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrpxw\"" Jul 19 07:43:38.674989 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.674954 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4znm8" Jul 19 07:43:38.795586 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.795496 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vkbl7"] Jul 19 07:43:38.799830 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:38.799806 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bff89a7_ac8c_48fe_badc_c78dd9f68166.slice/crio-08d643f21e514caf28535fb1ca0286269d5f5c96d74f5b4b7d1d73eb0bda1df6 WatchSource:0}: Error finding container 08d643f21e514caf28535fb1ca0286269d5f5c96d74f5b4b7d1d73eb0bda1df6: Status 404 returned error can't find the container with id 08d643f21e514caf28535fb1ca0286269d5f5c96d74f5b4b7d1d73eb0bda1df6 Jul 19 07:43:38.815760 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.815733 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xv94"] Jul 19 07:43:38.819049 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:38.819023 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2075b26b_ce71_46f3_a8a4_15ac53430240.slice/crio-458a6df2992b28789ef9ad8f48fb592578541d6191ee63968f41c9d14a90edf3 WatchSource:0}: Error finding container 458a6df2992b28789ef9ad8f48fb592578541d6191ee63968f41c9d14a90edf3: Status 404 returned error can't find the container with id 458a6df2992b28789ef9ad8f48fb592578541d6191ee63968f41c9d14a90edf3 Jul 19 07:43:38.834650 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:38.834624 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4znm8"] Jul 19 07:43:38.838039 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:38.838018 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34c1ae7_5ff5_4f73_8c5c_8ecf0078930c.slice/crio-c60c3cb8e3b4c674e7488e5ca3fbbabd4ce8678381d59ac6532d9bfca6adbf1f WatchSource:0}: Error finding container c60c3cb8e3b4c674e7488e5ca3fbbabd4ce8678381d59ac6532d9bfca6adbf1f: Status 404 returned error can't find the container with id c60c3cb8e3b4c674e7488e5ca3fbbabd4ce8678381d59ac6532d9bfca6adbf1f Jul 19 07:43:39.093245 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:39.093208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4znm8" event={"ID":"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c","Type":"ContainerStarted","Data":"c60c3cb8e3b4c674e7488e5ca3fbbabd4ce8678381d59ac6532d9bfca6adbf1f"} Jul 19 07:43:39.094298 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:39.094273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vkbl7" event={"ID":"8bff89a7-ac8c-48fe-badc-c78dd9f68166","Type":"ContainerStarted","Data":"08d643f21e514caf28535fb1ca0286269d5f5c96d74f5b4b7d1d73eb0bda1df6"} Jul 19 07:43:39.095374 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:39.095351 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xv94" event={"ID":"2075b26b-ce71-46f3-a8a4-15ac53430240","Type":"ContainerStarted","Data":"458a6df2992b28789ef9ad8f48fb592578541d6191ee63968f41c9d14a90edf3"} Jul 19 07:43:41.067508 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:41.067480 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7tll8" Jul 19 07:43:41.108174 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:41.108133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4znm8" event={"ID":"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c","Type":"ContainerStarted","Data":"7d2998fdf3fef4675ebcc6e6dba9b39520bc3bde7b0cfb9ff58fc89d3b08d4e4"} Jul 19 07:43:41.108349 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:41.108189 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4znm8" event={"ID":"f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c","Type":"ContainerStarted","Data":"7aad07f5154e0cb11f552a7d163da976c4c16080bfeae2827f2db03b33cbf96a"} Jul 19 07:43:41.122899 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:41.122852 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4znm8" podStartSLOduration=66.672537974 podStartE2EDuration="1m8.12283499s" totalImagesPullingTime="1.450297016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:41.122334582 +0000 UTC m=+68.901997889" watchObservedRunningTime="2026-07-19 07:43:41.12283499 +0000 UTC m=+68.902498368" Jul 19 07:43:44.119604 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.119565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xv94" event={"ID":"2075b26b-ce71-46f3-a8a4-15ac53430240","Type":"ContainerStarted","Data":"05230d01bd8362140bbd7e44b8a318bc08de4e1848f4f93a72610a244406ca20"} Jul 19 07:43:44.120095 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.119656 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:43:44.121031 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.121002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vkbl7" event={"ID":"8bff89a7-ac8c-48fe-badc-c78dd9f68166","Type":"ContainerStarted","Data":"e17e55b6c148590438b5eef14be95b5e704f319510e03ea6904c148e48f78e77"} Jul 19 07:43:44.136781 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.136740 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4xv94" podStartSLOduration=66.511269621 podStartE2EDuration="1m11.13672701s" totalImagesPullingTime="4.625457389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:44.136326722 +0000 UTC m=+71.915990045" watchObservedRunningTime="2026-07-19 07:43:44.13672701 +0000 UTC m=+71.916390324" Jul 19 07:43:44.149385 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.149346 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vkbl7" podStartSLOduration=66.49926109 podStartE2EDuration="1m11.149328398s" totalImagesPullingTime="4.650067308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:44.148851368 +0000 UTC m=+71.928514679" watchObservedRunningTime="2026-07-19 07:43:44.149328398 +0000 UTC m=+71.928991712" Jul 19 07:43:44.466171 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.466084 2574 patch_prober.go:28] interesting pod/image-registry-845bbc4585-824dm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 19 07:43:44.466171 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:44.466136 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-845bbc4585-824dm" podUID="bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 07:43:46.043593 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:46.043565 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-845bbc4585-824dm" Jul 19 07:43:47.828013 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.827978 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-brp9z"] Jul 19 07:43:47.830877 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.830861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.833846 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.833818 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 19 07:43:47.833959 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.833856 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 19 07:43:47.834308 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.834292 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 19 07:43:47.834951 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.834932 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 19 07:43:47.835032 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.834951 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 19 07:43:47.835032 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.834941 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 19 07:43:47.835032 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.834941 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5pdwm\"" Jul 19 07:43:47.880915 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.880885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-root\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.880915 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.880917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-textfile\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881108 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.880938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881108 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.880954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-tls\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881108 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.881019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-wtmp\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881108 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.881047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881108 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.881090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b337a1-f938-4bb2-97af-76675f05a4f5-metrics-client-ca\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881260 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.881120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xxc\" (UniqueName: \"kubernetes.io/projected/b2b337a1-f938-4bb2-97af-76675f05a4f5-kube-api-access-g7xxc\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.881260 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.881142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-sys\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982195 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b337a1-f938-4bb2-97af-76675f05a4f5-metrics-client-ca\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982290 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xxc\" (UniqueName: \"kubernetes.io/projected/b2b337a1-f938-4bb2-97af-76675f05a4f5-kube-api-access-g7xxc\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982290 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-sys\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982290 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-root\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-textfile\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-tls\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982358 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-sys\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-root\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-wtmp\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982706 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982706 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982493 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-wtmp\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982706 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:47.982509 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 19 07:43:47.982706 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:47.982603 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-tls podName:b2b337a1-f938-4bb2-97af-76675f05a4f5 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:48.482579155 +0000 UTC m=+76.262242462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-tls") pod "node-exporter-brp9z" (UID: "b2b337a1-f938-4bb2-97af-76675f05a4f5") : secret "node-exporter-tls" not found Jul 19 07:43:47.982706 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-textfile\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.982939 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.983008 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.982983 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b337a1-f938-4bb2-97af-76675f05a4f5-metrics-client-ca\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.984926 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.984904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:47.991025 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:47.990995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xxc\" (UniqueName: \"kubernetes.io/projected/b2b337a1-f938-4bb2-97af-76675f05a4f5-kube-api-access-g7xxc\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:48.487220 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.487169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-tls\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:48.489963 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.489942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2b337a1-f938-4bb2-97af-76675f05a4f5-node-exporter-tls\") pod \"node-exporter-brp9z\" (UID: \"b2b337a1-f938-4bb2-97af-76675f05a4f5\") " pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:48.740227 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.740155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-brp9z" Jul 19 07:43:48.751888 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:48.751858 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b337a1_f938_4bb2_97af_76675f05a4f5.slice/crio-66c2f746e41ef6e9c6c7e578902055470cd41a1b485a3f2d1d61e1d374efcf12 WatchSource:0}: Error finding container 66c2f746e41ef6e9c6c7e578902055470cd41a1b485a3f2d1d61e1d374efcf12: Status 404 returned error can't find the container with id 66c2f746e41ef6e9c6c7e578902055470cd41a1b485a3f2d1d61e1d374efcf12 Jul 19 07:43:48.855012 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.854972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:43:48.859843 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.859814 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.862793 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.862756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 19 07:43:48.862793 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.862756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 19 07:43:48.862978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.862760 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 19 07:43:48.863729 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.863664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 19 07:43:48.863931 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.863905 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 19 07:43:48.863931 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.863922 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrgpj\"" Jul 19 07:43:48.864090 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.864053 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 19 07:43:48.864216 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.864201 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 19 07:43:48.864285 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.864273 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 19 07:43:48.864753 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.864738 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 19 07:43:48.876770 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.876743 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:43:48.991993 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.991903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-web-config\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.991993 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.991939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.991993 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.991958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgxf\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-kube-api-access-rrgxf\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.991993 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.991977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.991993 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.991995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-out\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992281 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:48.992370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:48.992312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093499 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093645 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093645 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-out\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093943 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093943 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093943 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093943 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.093943 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:49.093937 2574 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 19 07:43:49.094174 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:43:49.094006 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls podName:c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:49.593984257 +0000 UTC m=+77.373647551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1") : secret "alertmanager-main-tls" not found Jul 19 07:43:49.094174 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-web-config\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.094174 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.094058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.094174 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.094090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgxf\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-kube-api-access-rrgxf\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.094587 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.093926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.095778 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.095413 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.096244 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.096216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.096923 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.096871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.097208 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.097177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.097292 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.097223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.097292 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.097223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.097370 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.097316 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-web-config\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.097405 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.097384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-out\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.098151 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.098133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.098412 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.098397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.101972 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.101952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgxf\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-kube-api-access-rrgxf\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.141729 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.141694 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brp9z" event={"ID":"b2b337a1-f938-4bb2-97af-76675f05a4f5","Type":"ContainerStarted","Data":"66c2f746e41ef6e9c6c7e578902055470cd41a1b485a3f2d1d61e1d374efcf12"} Jul 19 07:43:49.599383 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.599346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.602157 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.602132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.768671 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.768633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:43:49.924654 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:49.924623 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:43:49.927889 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:49.927858 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bc9c8f_9a0d_405f_89d3_ae6e1e9d68d1.slice/crio-48f56cbe7e39833395782e0bea0726e736d1f775de547ca1c5fc74485a8912ce WatchSource:0}: Error finding container 48f56cbe7e39833395782e0bea0726e736d1f775de547ca1c5fc74485a8912ce: Status 404 returned error can't find the container with id 48f56cbe7e39833395782e0bea0726e736d1f775de547ca1c5fc74485a8912ce Jul 19 07:43:50.146382 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.146279 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"48f56cbe7e39833395782e0bea0726e736d1f775de547ca1c5fc74485a8912ce"} Jul 19 07:43:50.147913 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.147890 2574 generic.go:356] "Generic (PLEG): container finished" podID="b2b337a1-f938-4bb2-97af-76675f05a4f5" containerID="98e69ac8918919722a9c09ea48c5942799ac81c636f57fc93445a15f42a4bbe6" exitCode=0 Jul 19 07:43:50.148041 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.147931 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brp9z" event={"ID":"b2b337a1-f938-4bb2-97af-76675f05a4f5","Type":"ContainerDied","Data":"98e69ac8918919722a9c09ea48c5942799ac81c636f57fc93445a15f42a4bbe6"} Jul 19 07:43:50.771753 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.771709 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b668f6f5-2jwcq"] Jul 19 07:43:50.775366 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.775342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.777774 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.777752 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vrwt7\"" Jul 19 07:43:50.777903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.777751 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 19 07:43:50.777903 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.777751 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 19 07:43:50.778213 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.778197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 19 07:43:50.778281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.778206 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 19 07:43:50.778281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.778271 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 19 07:43:50.778385 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.778292 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b3k1kc6ljb11k\"" Jul 19 07:43:50.787207 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.787185 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b668f6f5-2jwcq"] Jul 19 07:43:50.911090 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911289 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-metrics-client-ca\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911289 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911289 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-tls\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911441 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911441 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h267f\" (UniqueName: \"kubernetes.io/projected/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-kube-api-access-h267f\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911441 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-grpc-tls\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:50.911554 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:50.911460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012228 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-tls\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h267f\" (UniqueName: \"kubernetes.io/projected/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-kube-api-access-h267f\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-grpc-tls\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.012658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.012567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-metrics-client-ca\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.013567 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.013506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-metrics-client-ca\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.015124 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.015091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.015425 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.015401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-tls\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.015533 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.015452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.015533 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.015485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-grpc-tls\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.015614 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.015590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.015650 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.015595 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.019579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.019556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h267f\" (UniqueName: \"kubernetes.io/projected/48f3dac1-e06b-48b0-ad27-c069c3c41cd4-kube-api-access-h267f\") pod \"thanos-querier-6b668f6f5-2jwcq\" (UID: \"48f3dac1-e06b-48b0-ad27-c069c3c41cd4\") " pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.084018 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.083999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:51.154911 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.154617 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brp9z" event={"ID":"b2b337a1-f938-4bb2-97af-76675f05a4f5","Type":"ContainerStarted","Data":"7b0cb3fe8eb04c7935e8adfd8bc162e41e83ac4f2b4e1afa083ae4f908379eff"} Jul 19 07:43:51.154911 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.154717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brp9z" event={"ID":"b2b337a1-f938-4bb2-97af-76675f05a4f5","Type":"ContainerStarted","Data":"bd9b6d1233f9e70db6a0888eb8289814ffc1c128b942985eb650190b93a07097"} Jul 19 07:43:51.174528 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.173842 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-brp9z" podStartSLOduration=3.060490645 podStartE2EDuration="4.173819303s" totalImagesPullingTime="1.038266273s" totalInitContainerRuntime="75.062385ms" isStatefulPod=true podCreationTimestamp="2026-07-19 07:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:51.171890597 +0000 UTC m=+78.951553931" watchObservedRunningTime="2026-07-19 07:43:51.173819303 +0000 UTC m=+78.953482617" Jul 19 07:43:51.225641 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:51.225614 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b668f6f5-2jwcq"] Jul 19 07:43:52.092356 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.092319 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b4fd4859c-f5nlj"] Jul 19 07:43:52.095624 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.095598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.098027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.098004 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 19 07:43:52.098982 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.098960 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 19 07:43:52.098982 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.098972 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ltd58\"" Jul 19 07:43:52.099146 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.098970 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 19 07:43:52.099194 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.099146 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 19 07:43:52.099264 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.099246 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2a0mb10gqj6d6\"" Jul 19 07:43:52.104901 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.104877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b4fd4859c-f5nlj"] Jul 19 07:43:52.161576 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.161543 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a" exitCode=0 Jul 19 07:43:52.161758 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.161629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a"} Jul 19 07:43:52.162907 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.162884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"4720e6e14ce22d25a6dc6ca7668775b31203437ce0b9d8b3258d685450b65bfb"} Jul 19 07:43:52.222399 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.221792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.222399 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.221910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-audit-log\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.222399 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.221977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-secret-metrics-server-tls\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.222399 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.222006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b65n\" (UniqueName: \"kubernetes.io/projected/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-kube-api-access-8b65n\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.222774 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.222582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-secret-metrics-server-client-certs\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.222774 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.222633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-metrics-server-audit-profiles\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.222774 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.222745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-client-ca-bundle\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.323787 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-secret-metrics-server-client-certs\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.323787 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-metrics-server-audit-profiles\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-client-ca-bundle\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-audit-log\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-secret-metrics-server-tls\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.323954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b65n\" (UniqueName: \"kubernetes.io/projected/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-kube-api-access-8b65n\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324424 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.324382 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-audit-log\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324881 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.324863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-metrics-server-audit-profiles\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.324948 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.324924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.326946 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.326923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-secret-metrics-server-client-certs\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.327039 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.326941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-client-ca-bundle\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.327080 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.327062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-secret-metrics-server-tls\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.331716 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.331664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b65n\" (UniqueName: \"kubernetes.io/projected/37776d1e-46e0-45f6-9b4d-b5853a7b26ec-kube-api-access-8b65n\") pod \"metrics-server-5b4fd4859c-f5nlj\" (UID: \"37776d1e-46e0-45f6-9b4d-b5853a7b26ec\") " pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.407475 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.407390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:43:52.549560 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:52.549528 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b4fd4859c-f5nlj"] Jul 19 07:43:52.997700 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:52.997644 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37776d1e_46e0_45f6_9b4d_b5853a7b26ec.slice/crio-ce355cf52875266eade81ede67a3d8ffb7964894b0e7a2b3848cdd8777e93b08 WatchSource:0}: Error finding container ce355cf52875266eade81ede67a3d8ffb7964894b0e7a2b3848cdd8777e93b08: Status 404 returned error can't find the container with id ce355cf52875266eade81ede67a3d8ffb7964894b0e7a2b3848cdd8777e93b08 Jul 19 07:43:53.168140 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.168104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" event={"ID":"37776d1e-46e0-45f6-9b4d-b5853a7b26ec","Type":"ContainerStarted","Data":"ce355cf52875266eade81ede67a3d8ffb7964894b0e7a2b3848cdd8777e93b08"} Jul 19 07:43:53.173395 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.173364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"7523b9d9b6563e68e07077126b64078b2a0c111f2faad80a17b4df981ee62c3f"} Jul 19 07:43:53.971908 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.971867 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:43:53.976104 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.976077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:53.978630 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.978605 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 19 07:43:53.978898 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.978867 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hcd88n8nqoov\"" Jul 19 07:43:53.979129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979111 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 19 07:43:53.979220 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979147 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qx7cg\"" Jul 19 07:43:53.979220 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979159 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 19 07:43:53.979220 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 19 07:43:53.979366 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 19 07:43:53.979477 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979458 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 19 07:43:53.979534 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979483 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 19 07:43:53.979534 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979507 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 19 07:43:53.979534 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 19 07:43:53.979699 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979467 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 19 07:43:53.979759 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979704 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 19 07:43:53.979759 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979744 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 19 07:43:53.979863 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.979814 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 19 07:43:53.981426 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.981405 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 19 07:43:53.994989 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:53.994957 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:43:54.037384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-config\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.037566 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-web-config\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2md\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-kube-api-access-fs2md\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-config-out\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.038036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.037898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.138928 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.138880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.138928 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.138930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.138966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.138990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-web-config\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2md\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-kube-api-access-fs2md\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-config-out\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-config\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139427 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139900 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.139900 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.139476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.141545 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.141231 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.141545 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.141288 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.145723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.142781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.145723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.143378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.145723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.144788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.145723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.145408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-config-out\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.145723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.145597 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.146239 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.145808 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.146239 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.145816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.146239 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.145995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.147038 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.147009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.147871 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.147785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.147871 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.147802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.148304 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.148283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.148437 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.148392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.148736 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.148711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-web-config\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.149811 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.149789 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.149932 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.149910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.150425 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.150387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-config\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.153580 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.153556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2md\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-kube-api-access-fs2md\") pod \"prometheus-k8s-0\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.181036 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.180995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"26d5ae4a642a7f4dcbb51a24fcc072bf2e6745e44846313ddb747f1eff455a46"} Jul 19 07:43:54.181371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.181084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"87df1d6ecdbc43573e495f6e039946f45d7304e7649c28a2ce1bd81caea73ef1"} Jul 19 07:43:54.288917 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.288889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:43:54.456301 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:54.456211 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:43:54.730581 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:43:54.730554 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49338966_45dc_45b7_b1e7_b4982dc3018e.slice/crio-402bffec8663a102d1a68eb8ed1d55dd4bb4b03ee90f521795c87dec5fb610a8 WatchSource:0}: Error finding container 402bffec8663a102d1a68eb8ed1d55dd4bb4b03ee90f521795c87dec5fb610a8: Status 404 returned error can't find the container with id 402bffec8663a102d1a68eb8ed1d55dd4bb4b03ee90f521795c87dec5fb610a8 Jul 19 07:43:55.186754 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.186705 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" event={"ID":"37776d1e-46e0-45f6-9b4d-b5853a7b26ec","Type":"ContainerStarted","Data":"7a571afb38b2ac2b4b39eeb5b1c722140be215cbf148ccc14c724fb3a95307d7"} Jul 19 07:43:55.190379 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.190350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a"} Jul 19 07:43:55.190510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.190412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418"} Jul 19 07:43:55.190510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.190441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910"} Jul 19 07:43:55.190510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.190465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460"} Jul 19 07:43:55.190510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.190486 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4"} Jul 19 07:43:55.190510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.190512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerStarted","Data":"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988"} Jul 19 07:43:55.193465 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.193438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"4dee1cb49fb90faa5e4e9d435372315506b28c138c51ad596812cd74610fd644"} Jul 19 07:43:55.193581 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.193502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"bc9f3775d013191cfb95c41aeba0536436cae7964aae1804e367070be1561391"} Jul 19 07:43:55.193581 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.193529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" event={"ID":"48f3dac1-e06b-48b0-ad27-c069c3c41cd4","Type":"ContainerStarted","Data":"2c4f2f34e3b1138f89a519ae90a3322045f4e53f086a03b28d2573bc10461713"} Jul 19 07:43:55.193707 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.193585 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:43:55.194999 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.194973 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="b11dd201c8fe00ace43e13ce5e1f6264d451120df4d010b0d104c74689314c53" exitCode=0 Jul 19 07:43:55.195113 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.195043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"b11dd201c8fe00ace43e13ce5e1f6264d451120df4d010b0d104c74689314c53"} Jul 19 07:43:55.195113 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.195104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"402bffec8663a102d1a68eb8ed1d55dd4bb4b03ee90f521795c87dec5fb610a8"} Jul 19 07:43:55.203448 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.203413 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" podStartSLOduration=1.423512255 podStartE2EDuration="3.203399924s" totalImagesPullingTime="1.779887669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:55.203115063 +0000 UTC m=+82.982778377" watchObservedRunningTime="2026-07-19 07:43:55.203399924 +0000 UTC m=+82.983063238" Jul 19 07:43:55.252931 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.252828 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.977614095 podStartE2EDuration="7.252807674s" totalImagesPullingTime="3.258865353s" totalInitContainerRuntime="16.328226ms" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:55.251509143 +0000 UTC m=+83.031172458" watchObservedRunningTime="2026-07-19 07:43:55.252807674 +0000 UTC m=+83.032470991" Jul 19 07:43:55.274383 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:55.274325 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" podStartSLOduration=2.463906682 podStartE2EDuration="5.274303856s" totalImagesPullingTime="2.810397174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:55.27219236 +0000 UTC m=+83.051855694" watchObservedRunningTime="2026-07-19 07:43:55.274303856 +0000 UTC m=+83.053967168" Jul 19 07:43:58.212898 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:58.212866 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"d0e10378e2a3a58c02d33a5bde9da774ce85ac44d774c4f903e17de9e1b6a177"} Jul 19 07:43:59.220880 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.220843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"b34aeaa2d4b7eeda041f3ac251cc27514d39f83d5ace84fba048c52beff8edae"} Jul 19 07:43:59.221371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.220918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"79a617ecef48ba7f7f2aa29356018d146a8fbc470b584aaf22c16a75e3f3e526"} Jul 19 07:43:59.221371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.220959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"1f44aa2bf14b2c324e7a5f81cc6c0af5fb8913739b835ead48e2b2b17db63c11"} Jul 19 07:43:59.221371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.221001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"6509aeddae14dfbebc0f9c952d6722d431faa1aaeb7c504d01ee4337b3e29abd"} Jul 19 07:43:59.221371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.221053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerStarted","Data":"6181b2dfba7cc36d32e69b7da781207f1f3041aac7b7135c3435b9de5d9b7a3a"} Jul 19 07:43:59.246287 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.246196 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.329386701 podStartE2EDuration="6.246172393s" totalImagesPullingTime="2.89663763s" totalInitContainerRuntime="20.148062ms" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:59.244902611 +0000 UTC m=+87.024565925" watchObservedRunningTime="2026-07-19 07:43:59.246172393 +0000 UTC m=+87.025835709" Jul 19 07:43:59.289626 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:43:59.289587 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:44:01.205614 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:01.205588 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b668f6f5-2jwcq" Jul 19 07:44:12.408330 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:12.408291 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:44:12.408821 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:12.408355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:44:15.126961 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:15.126931 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4xv94" Jul 19 07:44:32.413450 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:32.413414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:44:32.417417 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:32.417392 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b4fd4859c-f5nlj" Jul 19 07:44:54.290071 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:54.290032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:44:54.309931 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:54.309898 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:44:54.458801 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:44:54.458776 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:08.019519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019482 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:45:08.020069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019800 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="alertmanager" containerID="cri-o://477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988" gracePeriod=120 Jul 19 07:45:08.020069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019865 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-web" containerID="cri-o://1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460" gracePeriod=120 Jul 19 07:45:08.020069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019904 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy" containerID="cri-o://7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910" gracePeriod=120 Jul 19 07:45:08.020069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019926 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="prom-label-proxy" containerID="cri-o://f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a" gracePeriod=120 Jul 19 07:45:08.020069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019921 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="config-reloader" containerID="cri-o://d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4" gracePeriod=120 Jul 19 07:45:08.020069 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.019871 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-metric" containerID="cri-o://68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418" gracePeriod=120 Jul 19 07:45:08.501548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501509 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a" exitCode=0 Jul 19 07:45:08.501548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501535 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910" exitCode=0 Jul 19 07:45:08.501548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501541 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4" exitCode=0 Jul 19 07:45:08.501548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501552 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988" exitCode=0 Jul 19 07:45:08.501832 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a"} Jul 19 07:45:08.501832 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910"} Jul 19 07:45:08.501832 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4"} Jul 19 07:45:08.501832 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:08.501755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988"} Jul 19 07:45:09.255658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.255634 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.301691 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301658 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.301859 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-main-db\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.301859 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-metrics-client-ca\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.301859 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.301859 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301811 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-trusted-ca-bundle\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.301859 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301846 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-volume\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301887 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301912 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-tls-assets\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301955 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrgxf\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-kube-api-access-rrgxf\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.301982 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-out\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.302024 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-cluster-tls-config\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.302050 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302121 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.302112 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-web-config\") pod \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\" (UID: \"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1\") " Jul 19 07:45:09.302519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.302484 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:09.303061 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.302654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:45:09.303383 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.303344 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:09.307569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.307088 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.307569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.307354 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.307569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.307478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:45:09.307569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.307555 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-kube-api-access-rrgxf" (OuterVolumeSpecName: "kube-api-access-rrgxf") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "kube-api-access-rrgxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:45:09.307920 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.307815 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.308313 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.308091 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.308494 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.308448 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-out" (OuterVolumeSpecName: "config-out") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:45:09.309299 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.309278 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-volume" (OuterVolumeSpecName: "config-volume") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.310916 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.310778 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.318206 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.318186 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-web-config" (OuterVolumeSpecName: "web-config") pod "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" (UID: "c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:09.403456 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403413 2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-volume\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403456 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403450 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403483 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-tls-assets\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403495 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrgxf\" (UniqueName: \"kubernetes.io/projected/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-kube-api-access-rrgxf\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403508 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-config-out\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403519 2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-cluster-tls-config\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403531 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403542 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-web-config\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403554 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-main-tls\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403564 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-main-db\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403575 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-metrics-client-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403586 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.403739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.403598 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:09.508709 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508659 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418" exitCode=0 Jul 19 07:45:09.508709 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508700 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerID="1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460" exitCode=0 Jul 19 07:45:09.508905 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508707 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418"} Jul 19 07:45:09.508905 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508750 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.508905 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508779 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460"} Jul 19 07:45:09.508905 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1","Type":"ContainerDied","Data":"48f56cbe7e39833395782e0bea0726e736d1f775de547ca1c5fc74485a8912ce"} Jul 19 07:45:09.508905 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.508838 2574 scope.go:117] "RemoveContainer" containerID="f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a" Jul 19 07:45:09.516960 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.516793 2574 scope.go:117] "RemoveContainer" containerID="68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418" Jul 19 07:45:09.524021 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.524002 2574 scope.go:117] "RemoveContainer" containerID="7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910" Jul 19 07:45:09.531346 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.531321 2574 scope.go:117] "RemoveContainer" containerID="1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460" Jul 19 07:45:09.531647 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.531624 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:45:09.536279 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.536256 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:45:09.539002 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.538983 2574 scope.go:117] "RemoveContainer" containerID="d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4" Jul 19 07:45:09.545992 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.545969 2574 scope.go:117] "RemoveContainer" containerID="477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988" Jul 19 07:45:09.553506 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.553488 2574 scope.go:117] "RemoveContainer" containerID="2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a" Jul 19 07:45:09.560739 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.560639 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561029 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="init-config-reloader" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561048 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="init-config-reloader" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561063 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-web" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561071 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-web" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561080 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="prom-label-proxy" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561088 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="prom-label-proxy" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561099 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561107 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561136 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="config-reloader" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561145 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="config-reloader" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561161 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="alertmanager" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561168 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="alertmanager" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561186 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-metric" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561194 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-metric" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561253 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-metric" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561264 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="prom-label-proxy" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561273 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="config-reloader" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561283 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="alertmanager" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561292 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy-web" Jul 19 07:45:09.561548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.561300 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" containerName="kube-rbac-proxy" Jul 19 07:45:09.562752 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.562729 2574 scope.go:117] "RemoveContainer" containerID="f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a" Jul 19 07:45:09.563067 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.563037 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a\": container with ID starting with f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a not found: ID does not exist" containerID="f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a" Jul 19 07:45:09.563128 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563076 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a"} err="failed to get container status \"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a\": rpc error: code = NotFound desc = could not find container \"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a\": container with ID starting with f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a not found: ID does not exist" Jul 19 07:45:09.563128 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563108 2574 scope.go:117] "RemoveContainer" containerID="68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418" Jul 19 07:45:09.563379 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.563358 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418\": container with ID starting with 68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418 not found: ID does not exist" containerID="68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418" Jul 19 07:45:09.563430 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563385 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418"} err="failed to get container status \"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418\": rpc error: code = NotFound desc = could not find container \"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418\": container with ID starting with 68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418 not found: ID does not exist" Jul 19 07:45:09.563430 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563403 2574 scope.go:117] "RemoveContainer" containerID="7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910" Jul 19 07:45:09.563658 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.563643 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910\": container with ID starting with 7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910 not found: ID does not exist" containerID="7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910" Jul 19 07:45:09.563735 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563662 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910"} err="failed to get container status \"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910\": rpc error: code = NotFound desc = could not find container \"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910\": container with ID starting with 7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910 not found: ID does not exist" Jul 19 07:45:09.563735 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563693 2574 scope.go:117] "RemoveContainer" containerID="1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460" Jul 19 07:45:09.563965 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.563937 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460\": container with ID starting with 1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460 not found: ID does not exist" containerID="1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460" Jul 19 07:45:09.564019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563974 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460"} err="failed to get container status \"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460\": rpc error: code = NotFound desc = could not find container \"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460\": container with ID starting with 1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460 not found: ID does not exist" Jul 19 07:45:09.564019 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.563995 2574 scope.go:117] "RemoveContainer" containerID="d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4" Jul 19 07:45:09.564226 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.564210 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4\": container with ID starting with d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4 not found: ID does not exist" containerID="d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4" Jul 19 07:45:09.564271 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564231 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4"} err="failed to get container status \"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4\": rpc error: code = NotFound desc = could not find container \"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4\": container with ID starting with d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4 not found: ID does not exist" Jul 19 07:45:09.564271 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564246 2574 scope.go:117] "RemoveContainer" containerID="477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988" Jul 19 07:45:09.564500 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.564480 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988\": container with ID starting with 477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988 not found: ID does not exist" containerID="477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988" Jul 19 07:45:09.564555 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564506 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988"} err="failed to get container status \"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988\": rpc error: code = NotFound desc = could not find container \"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988\": container with ID starting with 477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988 not found: ID does not exist" Jul 19 07:45:09.564555 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564518 2574 scope.go:117] "RemoveContainer" containerID="2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a" Jul 19 07:45:09.564758 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:45:09.564739 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a\": container with ID starting with 2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a not found: ID does not exist" containerID="2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a" Jul 19 07:45:09.564791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564763 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a"} err="failed to get container status \"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a\": rpc error: code = NotFound desc = could not find container \"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a\": container with ID starting with 2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a not found: ID does not exist" Jul 19 07:45:09.564791 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564780 2574 scope.go:117] "RemoveContainer" containerID="f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a" Jul 19 07:45:09.564957 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564941 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a"} err="failed to get container status \"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a\": rpc error: code = NotFound desc = could not find container \"f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a\": container with ID starting with f75562cca742f0fcdfa080ecf4912a3774b7561863c8962eee575e3204acd20a not found: ID does not exist" Jul 19 07:45:09.565000 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.564957 2574 scope.go:117] "RemoveContainer" containerID="68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418" Jul 19 07:45:09.565138 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565124 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418"} err="failed to get container status \"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418\": rpc error: code = NotFound desc = could not find container \"68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418\": container with ID starting with 68cbfe51325ad07dcfc58e83e1eb200cbb5a7d087dc0158a98363b2c6a6aa418 not found: ID does not exist" Jul 19 07:45:09.565173 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565139 2574 scope.go:117] "RemoveContainer" containerID="7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910" Jul 19 07:45:09.565296 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565281 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910"} err="failed to get container status \"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910\": rpc error: code = NotFound desc = could not find container \"7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910\": container with ID starting with 7dceeccd4a1b0df46b3cf1b35e5322f6f8c0e1b76b3d73878dd8a9734c376910 not found: ID does not exist" Jul 19 07:45:09.565332 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565299 2574 scope.go:117] "RemoveContainer" containerID="1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460" Jul 19 07:45:09.565486 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565468 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460"} err="failed to get container status \"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460\": rpc error: code = NotFound desc = could not find container \"1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460\": container with ID starting with 1f555fa6d5517dfd88c86a84ff7fa855e5312d2e6bc53fef315964dd43af7460 not found: ID does not exist" Jul 19 07:45:09.565526 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565488 2574 scope.go:117] "RemoveContainer" containerID="d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4" Jul 19 07:45:09.565711 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565670 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4"} err="failed to get container status \"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4\": rpc error: code = NotFound desc = could not find container \"d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4\": container with ID starting with d489892bd7ecf47b6d47bf4d6829f6598fdaa1706039e0c537aff5434584c0c4 not found: ID does not exist" Jul 19 07:45:09.565753 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565713 2574 scope.go:117] "RemoveContainer" containerID="477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988" Jul 19 07:45:09.565938 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565920 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988"} err="failed to get container status \"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988\": rpc error: code = NotFound desc = could not find container \"477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988\": container with ID starting with 477e5a0dc14d53f00df1dc70bff902d0f0e8e94b10a0b5ad51691a6996613988 not found: ID does not exist" Jul 19 07:45:09.565976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.565939 2574 scope.go:117] "RemoveContainer" containerID="2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a" Jul 19 07:45:09.566161 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.566143 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a"} err="failed to get container status \"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a\": rpc error: code = NotFound desc = could not find container \"2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a\": container with ID starting with 2cf1cc2a6ad1d2002e046e2be26e50cd94d9f849df8061ce94fbd1fe2325055a not found: ID does not exist" Jul 19 07:45:09.566949 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.566937 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.569342 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.569326 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 19 07:45:09.569440 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.569325 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 19 07:45:09.569440 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.569326 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 19 07:45:09.569702 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.569667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 19 07:45:09.569806 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.569744 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 19 07:45:09.569806 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.569760 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrgpj\"" Jul 19 07:45:09.570589 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.570508 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 19 07:45:09.570589 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.570521 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 19 07:45:09.570851 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.570835 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 19 07:45:09.577613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.577468 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 19 07:45:09.579741 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.579719 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:45:09.605622 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605622 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgbmj\" (UniqueName: \"kubernetes.io/projected/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-kube-api-access-fgbmj\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-web-config\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-config-out\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.605882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.606178 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.606178 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.606178 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.605988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.606178 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.606034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.706909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.706953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-config-out\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.706974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.707596 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.707468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710189 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-config-out\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710326 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710326 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710326 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbmj\" (UniqueName: \"kubernetes.io/projected/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-kube-api-access-fgbmj\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710324 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-web-config\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710754 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.710961 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-config-volume\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.711027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.710943 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.711244 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.711225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.712436 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.712419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-web-config\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.718047 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.718025 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbmj\" (UniqueName: \"kubernetes.io/projected/0b2ded24-eb51-4f3c-bb13-1da328f80cd3-kube-api-access-fgbmj\") pod \"alertmanager-main-0\" (UID: \"0b2ded24-eb51-4f3c-bb13-1da328f80cd3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:09.879882 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:09.879845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 07:45:10.013787 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:10.013749 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 07:45:10.017147 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:45:10.017119 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2ded24_eb51_4f3c_bb13_1da328f80cd3.slice/crio-0a9cbb26d047d3756c72ee94d1c999afb8d9723854a1acef2963041bde85f573 WatchSource:0}: Error finding container 0a9cbb26d047d3756c72ee94d1c999afb8d9723854a1acef2963041bde85f573: Status 404 returned error can't find the container with id 0a9cbb26d047d3756c72ee94d1c999afb8d9723854a1acef2963041bde85f573 Jul 19 07:45:10.514598 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:10.514566 2574 generic.go:356] "Generic (PLEG): container finished" podID="0b2ded24-eb51-4f3c-bb13-1da328f80cd3" containerID="4ed579888f91800f8c94347e376c6a49b6cc42b2f8c693ae596f50f6fbd0e0df" exitCode=0 Jul 19 07:45:10.515150 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:10.514610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerDied","Data":"4ed579888f91800f8c94347e376c6a49b6cc42b2f8c693ae596f50f6fbd0e0df"} Jul 19 07:45:10.515150 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:10.514662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"0a9cbb26d047d3756c72ee94d1c999afb8d9723854a1acef2963041bde85f573"} Jul 19 07:45:10.732384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:10.732356 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1" path="/var/lib/kubelet/pods/c0bc9c8f-9a0d-405f-89d3-ae6e1e9d68d1/volumes" Jul 19 07:45:11.523182 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.523144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"7a14df0401829bcd16935fcb3130db2cc79d724ed5c69d0e6cef6862e12ef393"} Jul 19 07:45:11.523565 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.523218 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"5c0ba7ec8a73f3f274bf553a3722a5d81ddecfa6f20b970d2e8ca040433e0555"} Jul 19 07:45:11.523565 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.523244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"9b86d8da0050ae7f22f7c099f812474af99b90e1690989f76e6e74e2942c68b6"} Jul 19 07:45:11.523565 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.523270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"5d4847e7c79b83dc02f43d60225172e0d692c363945855b11d8d19d29b57738b"} Jul 19 07:45:11.523565 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.523294 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"190efa4eb70f9c4fb8d3243cf2c0775d5944e95a2ac01a7da02d77efa6ca8148"} Jul 19 07:45:11.523565 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.523316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0b2ded24-eb51-4f3c-bb13-1da328f80cd3","Type":"ContainerStarted","Data":"8373312bddbfdb0bc32cbed76e92600698f8a9e858eebd067c99896196447821"} Jul 19 07:45:11.549340 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:11.549280 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5333284000000003 podStartE2EDuration="2.549258648s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.930248ms" isStatefulPod=false podCreationTimestamp="2026-07-19 07:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:45:11.548249711 +0000 UTC m=+159.327913025" watchObservedRunningTime="2026-07-19 07:45:11.549258648 +0000 UTC m=+159.328921965" Jul 19 07:45:12.322105 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322074 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:45:12.322471 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322402 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="prometheus" containerID="cri-o://d0e10378e2a3a58c02d33a5bde9da774ce85ac44d774c4f903e17de9e1b6a177" gracePeriod=600 Jul 19 07:45:12.322471 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322424 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b34aeaa2d4b7eeda041f3ac251cc27514d39f83d5ace84fba048c52beff8edae" gracePeriod=600 Jul 19 07:45:12.322652 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322454 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="thanos-sidecar" containerID="cri-o://6509aeddae14dfbebc0f9c952d6722d431faa1aaeb7c504d01ee4337b3e29abd" gracePeriod=600 Jul 19 07:45:12.322652 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322468 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-web" containerID="cri-o://1f44aa2bf14b2c324e7a5f81cc6c0af5fb8913739b835ead48e2b2b17db63c11" gracePeriod=600 Jul 19 07:45:12.322652 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322464 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="config-reloader" containerID="cri-o://6181b2dfba7cc36d32e69b7da781207f1f3041aac7b7135c3435b9de5d9b7a3a" gracePeriod=600 Jul 19 07:45:12.322652 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.322408 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy" containerID="cri-o://79a617ecef48ba7f7f2aa29356018d146a8fbc470b584aaf22c16a75e3f3e526" gracePeriod=600 Jul 19 07:45:12.531723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531687 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="b34aeaa2d4b7eeda041f3ac251cc27514d39f83d5ace84fba048c52beff8edae" exitCode=0 Jul 19 07:45:12.531723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531721 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="79a617ecef48ba7f7f2aa29356018d146a8fbc470b584aaf22c16a75e3f3e526" exitCode=0 Jul 19 07:45:12.531723 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"b34aeaa2d4b7eeda041f3ac251cc27514d39f83d5ace84fba048c52beff8edae"} Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531730 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="1f44aa2bf14b2c324e7a5f81cc6c0af5fb8913739b835ead48e2b2b17db63c11" exitCode=0 Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"79a617ecef48ba7f7f2aa29356018d146a8fbc470b584aaf22c16a75e3f3e526"} Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531856 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"1f44aa2bf14b2c324e7a5f81cc6c0af5fb8913739b835ead48e2b2b17db63c11"} Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531814 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="6509aeddae14dfbebc0f9c952d6722d431faa1aaeb7c504d01ee4337b3e29abd" exitCode=0 Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531892 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="6181b2dfba7cc36d32e69b7da781207f1f3041aac7b7135c3435b9de5d9b7a3a" exitCode=0 Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531896 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"6509aeddae14dfbebc0f9c952d6722d431faa1aaeb7c504d01ee4337b3e29abd"} Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"6181b2dfba7cc36d32e69b7da781207f1f3041aac7b7135c3435b9de5d9b7a3a"} Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531900 2574 generic.go:356] "Generic (PLEG): container finished" podID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerID="d0e10378e2a3a58c02d33a5bde9da774ce85ac44d774c4f903e17de9e1b6a177" exitCode=0 Jul 19 07:45:12.532133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.531986 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"d0e10378e2a3a58c02d33a5bde9da774ce85ac44d774c4f903e17de9e1b6a177"} Jul 19 07:45:12.567652 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.567626 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:12.637133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637095 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-config-out\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs2md\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-kube-api-access-fs2md\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637363 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637183 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-tls\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637363 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637363 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637238 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-kubelet-serving-ca-bundle\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637363 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637261 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-web-config\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637511 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-2\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637511 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637421 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-thanos-prometheus-http-client-file\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637511 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637462 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-trusted-ca-bundle\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637511 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637488 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-serving-certs-ca-bundle\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637521 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-metrics-client-certs\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637545 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-1\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637577 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-grpc-tls\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637612 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637790 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637640 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-0\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637790 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637694 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-kube-rbac-proxy\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637790 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637727 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-db\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637913 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637790 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-tls-assets\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637913 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637817 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-config\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.637913 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.637866 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-metrics-client-ca\") pod \"49338966-45dc-45b7-b1e7-b4982dc3018e\" (UID: \"49338966-45dc-45b7-b1e7-b4982dc3018e\") " Jul 19 07:45:12.639639 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.639481 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.639639 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.639618 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.639863 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.639768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.640196 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.640013 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.640196 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.640046 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.640362 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.640270 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-config-out" (OuterVolumeSpecName: "config-out") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:45:12.641281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.640986 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.641281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.641073 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:45:12.641696 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.641638 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.642249 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.642201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:45:12.642430 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.642404 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-kube-api-access-fs2md" (OuterVolumeSpecName: "kube-api-access-fs2md") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "kube-api-access-fs2md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:45:12.642902 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.642870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.642999 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.642920 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.643384 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.643349 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.643786 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.643760 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.643967 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.643938 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.644165 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.644149 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.644771 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.644732 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-config" (OuterVolumeSpecName: "config") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.644848 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.644822 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:45:12.652696 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.652664 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-web-config" (OuterVolumeSpecName: "web-config") pod "49338966-45dc-45b7-b1e7-b4982dc3018e" (UID: "49338966-45dc-45b7-b1e7-b4982dc3018e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:45:12.738901 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738879 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-tls-assets\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.738901 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738902 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-config\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738912 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-metrics-client-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738927 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-config-out\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738947 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2md\" (UniqueName: \"kubernetes.io/projected/49338966-45dc-45b7-b1e7-b4982dc3018e-kube-api-access-fs2md\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738957 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738967 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738976 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738984 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-web-config\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.738993 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739001 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739011 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739027 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739019 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739030 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-metrics-client-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739043 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739052 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-grpc-tls\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739060 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739077 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739085 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/49338966-45dc-45b7-b1e7-b4982dc3018e-secret-kube-rbac-proxy\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:12.739343 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:12.739093 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/49338966-45dc-45b7-b1e7-b4982dc3018e-prometheus-k8s-db\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:45:13.539412 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.539328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"49338966-45dc-45b7-b1e7-b4982dc3018e","Type":"ContainerDied","Data":"402bffec8663a102d1a68eb8ed1d55dd4bb4b03ee90f521795c87dec5fb610a8"} Jul 19 07:45:13.539412 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.539407 2574 scope.go:117] "RemoveContainer" containerID="b34aeaa2d4b7eeda041f3ac251cc27514d39f83d5ace84fba048c52beff8edae" Jul 19 07:45:13.539867 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.539336 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.551642 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.551604 2574 scope.go:117] "RemoveContainer" containerID="79a617ecef48ba7f7f2aa29356018d146a8fbc470b584aaf22c16a75e3f3e526" Jul 19 07:45:13.559382 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.559357 2574 scope.go:117] "RemoveContainer" containerID="1f44aa2bf14b2c324e7a5f81cc6c0af5fb8913739b835ead48e2b2b17db63c11" Jul 19 07:45:13.563333 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.563311 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:45:13.567179 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.567163 2574 scope.go:117] "RemoveContainer" containerID="6509aeddae14dfbebc0f9c952d6722d431faa1aaeb7c504d01ee4337b3e29abd" Jul 19 07:45:13.567381 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.567359 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:45:13.574093 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.574075 2574 scope.go:117] "RemoveContainer" containerID="6181b2dfba7cc36d32e69b7da781207f1f3041aac7b7135c3435b9de5d9b7a3a" Jul 19 07:45:13.580887 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.580862 2574 scope.go:117] "RemoveContainer" containerID="d0e10378e2a3a58c02d33a5bde9da774ce85ac44d774c4f903e17de9e1b6a177" Jul 19 07:45:13.588438 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.588418 2574 scope.go:117] "RemoveContainer" containerID="b11dd201c8fe00ace43e13ce5e1f6264d451120df4d010b0d104c74689314c53" Jul 19 07:45:13.592245 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592220 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:45:13.592663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592645 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="prometheus" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592665 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="prometheus" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592692 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="init-config-reloader" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592702 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="init-config-reloader" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592724 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="config-reloader" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592730 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="config-reloader" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592740 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy" Jul 19 07:45:13.592749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592745 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592754 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-web" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592760 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-web" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592766 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-thanos" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592772 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-thanos" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592784 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="thanos-sidecar" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592792 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="thanos-sidecar" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592857 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-web" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592870 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="config-reloader" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592876 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="thanos-sidecar" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592882 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy-thanos" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592889 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="prometheus" Jul 19 07:45:13.592976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.592895 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" containerName="kube-rbac-proxy" Jul 19 07:45:13.597702 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.597669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.599919 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.599899 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 19 07:45:13.599919 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.599900 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 19 07:45:13.600063 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.599982 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 19 07:45:13.600219 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.600204 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 19 07:45:13.600565 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.600551 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 19 07:45:13.600619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.600550 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 19 07:45:13.600619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.600599 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3hcd88n8nqoov\"" Jul 19 07:45:13.600619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.600609 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 19 07:45:13.601086 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.601064 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 19 07:45:13.601349 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.601333 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qx7cg\"" Jul 19 07:45:13.601349 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.601341 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 19 07:45:13.601495 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.601432 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 19 07:45:13.601580 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.601542 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 19 07:45:13.601916 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.601899 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 19 07:45:13.603520 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.603499 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 19 07:45:13.606328 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.606310 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 19 07:45:13.616492 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.616466 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:45:13.646451 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-config\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646451 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646663 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646727 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d6p\" (UniqueName: \"kubernetes.io/projected/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-kube-api-access-p9d6p\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.646978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.646967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.647127 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.647007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.647127 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.647033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.647127 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.647049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748052 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748052 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748239 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748239 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-config\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748306 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748306 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748391 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748441 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748503 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748503 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748602 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748602 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748602 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d6p\" (UniqueName: \"kubernetes.io/projected/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-kube-api-access-p9d6p\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.748813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.749096 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.749096 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.748883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.749096 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.749048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.749985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.749637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.749985 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.749695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.750469 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.750441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.751505 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.751478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.751596 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.751478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-config\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.751795 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.751772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.752138 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.752111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.752212 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.752189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.752212 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.752199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.752413 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.752388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.753211 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.753154 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.753504 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.753480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.753583 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.753517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.754516 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.754492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.754780 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.754762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.754978 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.754958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.755047 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.755018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.759707 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.759662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d6p\" (UniqueName: \"kubernetes.io/projected/fae95004-cea2-4710-a5b6-d3bb5c60b5dc-kube-api-access-p9d6p\") pod \"prometheus-k8s-0\" (UID: \"fae95004-cea2-4710-a5b6-d3bb5c60b5dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:13.906908 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:13.906870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:45:14.044620 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:14.044590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 07:45:14.047856 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:45:14.047823 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae95004_cea2_4710_a5b6_d3bb5c60b5dc.slice/crio-263af6b75544b715b51bde7a50902e1ab29020e424ce59e15eca9f0248e129dc WatchSource:0}: Error finding container 263af6b75544b715b51bde7a50902e1ab29020e424ce59e15eca9f0248e129dc: Status 404 returned error can't find the container with id 263af6b75544b715b51bde7a50902e1ab29020e424ce59e15eca9f0248e129dc Jul 19 07:45:14.545231 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:14.545195 2574 generic.go:356] "Generic (PLEG): container finished" podID="fae95004-cea2-4710-a5b6-d3bb5c60b5dc" containerID="5a683e7bedd20a7d981db05de889cf5c8a80e758fabce2402fbc7cc37ce8867d" exitCode=0 Jul 19 07:45:14.545621 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:14.545244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerDied","Data":"5a683e7bedd20a7d981db05de889cf5c8a80e758fabce2402fbc7cc37ce8867d"} Jul 19 07:45:14.545621 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:14.545300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"263af6b75544b715b51bde7a50902e1ab29020e424ce59e15eca9f0248e129dc"} Jul 19 07:45:14.730507 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:14.730470 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49338966-45dc-45b7-b1e7-b4982dc3018e" path="/var/lib/kubelet/pods/49338966-45dc-45b7-b1e7-b4982dc3018e/volumes" Jul 19 07:45:15.553483 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.553448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"7432ee81ac7e70a038f87374169bdbaae81215f95a2ad6829636646047d33d4c"} Jul 19 07:45:15.553875 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.553516 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"a6b4037b7d1ad89452a375016072f37102ccf1fae39197f61006b8045edff027"} Jul 19 07:45:15.553875 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.553543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"19006621e2039b0606ef775b0c6f56941d514b128fee136616ac8651869406ad"} Jul 19 07:45:15.553875 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.553567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"af78834a3901b012e02577ba1bf3893148969d26754c0edfd15dde852679a2fc"} Jul 19 07:45:15.553875 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.553596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"3838978f51216e61fca6991ac0bc9c39c67fe10fa4e47d7e52421e205964f3d5"} Jul 19 07:45:15.553875 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.553619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae95004-cea2-4710-a5b6-d3bb5c60b5dc","Type":"ContainerStarted","Data":"311e7cf870a8bdde02c37225f96e5a16047085023f70504dac4c9ee3a7aec050"} Jul 19 07:45:15.580867 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:15.580818 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.563064092 podStartE2EDuration="2.580796428s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.732336ms" isStatefulPod=false podCreationTimestamp="2026-07-19 07:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:45:15.57839616 +0000 UTC m=+163.358059473" watchObservedRunningTime="2026-07-19 07:45:15.580796428 +0000 UTC m=+163.360459741" Jul 19 07:45:18.907964 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:45:18.907910 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:46:13.907534 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:46:13.907494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:46:13.923569 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:46:13.923546 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:46:14.785635 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:46:14.785611 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 07:47:32.609445 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:47:32.609416 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 19 07:48:34.289419 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.289378 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq"] Jul 19 07:48:34.292489 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.292469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.298551 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.298524 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 19 07:48:34.298551 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.298540 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 19 07:48:34.298779 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.298554 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 19 07:48:34.298779 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.298569 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8c278\"" Jul 19 07:48:34.298779 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.298612 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 19 07:48:34.320262 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.320232 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq"] Jul 19 07:48:34.433178 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.433144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cff2e13b-0644-4cc6-be23-f49ed171dde0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.433355 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.433189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff2e13b-0644-4cc6-be23-f49ed171dde0-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.433355 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.433217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff2e13b-0644-4cc6-be23-f49ed171dde0-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.433355 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.433252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff2e13b-0644-4cc6-be23-f49ed171dde0-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.433355 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.433314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvnj\" (UniqueName: \"kubernetes.io/projected/cff2e13b-0644-4cc6-be23-f49ed171dde0-kube-api-access-kjvnj\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.534560 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.534528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff2e13b-0644-4cc6-be23-f49ed171dde0-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.534560 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.534561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff2e13b-0644-4cc6-be23-f49ed171dde0-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.534874 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.534605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvnj\" (UniqueName: \"kubernetes.io/projected/cff2e13b-0644-4cc6-be23-f49ed171dde0-kube-api-access-kjvnj\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.534874 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.534654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cff2e13b-0644-4cc6-be23-f49ed171dde0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.534874 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.534723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff2e13b-0644-4cc6-be23-f49ed171dde0-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.535035 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.535013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff2e13b-0644-4cc6-be23-f49ed171dde0-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.537157 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.537127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff2e13b-0644-4cc6-be23-f49ed171dde0-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.537299 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.537240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cff2e13b-0644-4cc6-be23-f49ed171dde0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.537394 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.537375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff2e13b-0644-4cc6-be23-f49ed171dde0-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.542874 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.542825 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvnj\" (UniqueName: \"kubernetes.io/projected/cff2e13b-0644-4cc6-be23-f49ed171dde0-kube-api-access-kjvnj\") pod \"opendatahub-operator-controller-manager-789467ff8c-2wlfq\" (UID: \"cff2e13b-0644-4cc6-be23-f49ed171dde0\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.608177 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.608147 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:34.751662 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.751625 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq"] Jul 19 07:48:34.755728 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:48:34.755698 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff2e13b_0644_4cc6_be23_f49ed171dde0.slice/crio-7c35eea0232d664c15336d5b83b4100874ad8cbf7509bde80647b6a0e10c42f2 WatchSource:0}: Error finding container 7c35eea0232d664c15336d5b83b4100874ad8cbf7509bde80647b6a0e10c42f2: Status 404 returned error can't find the container with id 7c35eea0232d664c15336d5b83b4100874ad8cbf7509bde80647b6a0e10c42f2 Jul 19 07:48:34.757449 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.757427 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 07:48:34.760920 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.760891 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-747d9ff956-lchls"] Jul 19 07:48:34.764964 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.764949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.768556 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.768529 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tkwth\"" Jul 19 07:48:34.768659 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.768529 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 19 07:48:34.768659 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.768601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 19 07:48:34.768659 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.768578 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 19 07:48:34.768848 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.768711 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 19 07:48:34.768900 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.768885 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 19 07:48:34.781980 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.781957 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-747d9ff956-lchls"] Jul 19 07:48:34.837922 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.837886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af121060-cd61-494e-8b87-13ba67a89698-cert\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.838083 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.837941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af121060-cd61-494e-8b87-13ba67a89698-metrics-cert\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.838083 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.837975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xgw\" (UniqueName: \"kubernetes.io/projected/af121060-cd61-494e-8b87-13ba67a89698-kube-api-access-b9xgw\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.838083 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.838051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af121060-cd61-494e-8b87-13ba67a89698-manager-config\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.939395 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.939359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af121060-cd61-494e-8b87-13ba67a89698-manager-config\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.939572 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.939406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af121060-cd61-494e-8b87-13ba67a89698-cert\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.939572 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.939434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af121060-cd61-494e-8b87-13ba67a89698-metrics-cert\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.939572 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.939453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xgw\" (UniqueName: \"kubernetes.io/projected/af121060-cd61-494e-8b87-13ba67a89698-kube-api-access-b9xgw\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.940201 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.940138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/af121060-cd61-494e-8b87-13ba67a89698-manager-config\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.942135 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.942111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af121060-cd61-494e-8b87-13ba67a89698-metrics-cert\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.942222 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.942115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af121060-cd61-494e-8b87-13ba67a89698-cert\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:34.947959 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:34.947942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xgw\" (UniqueName: \"kubernetes.io/projected/af121060-cd61-494e-8b87-13ba67a89698-kube-api-access-b9xgw\") pod \"lws-controller-manager-747d9ff956-lchls\" (UID: \"af121060-cd61-494e-8b87-13ba67a89698\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:35.074492 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:35.074465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:35.215418 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:35.215389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-747d9ff956-lchls"] Jul 19 07:48:35.217813 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:48:35.217785 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf121060_cd61_494e_8b87_13ba67a89698.slice/crio-c2f97c9e9ef232872db84e9bb6186f2e63588f8f9fc5eabf165e77a34cd1818f WatchSource:0}: Error finding container c2f97c9e9ef232872db84e9bb6186f2e63588f8f9fc5eabf165e77a34cd1818f: Status 404 returned error can't find the container with id c2f97c9e9ef232872db84e9bb6186f2e63588f8f9fc5eabf165e77a34cd1818f Jul 19 07:48:35.275996 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:35.275962 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" event={"ID":"cff2e13b-0644-4cc6-be23-f49ed171dde0","Type":"ContainerStarted","Data":"7c35eea0232d664c15336d5b83b4100874ad8cbf7509bde80647b6a0e10c42f2"} Jul 19 07:48:35.277002 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:35.276979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" event={"ID":"af121060-cd61-494e-8b87-13ba67a89698","Type":"ContainerStarted","Data":"c2f97c9e9ef232872db84e9bb6186f2e63588f8f9fc5eabf165e77a34cd1818f"} Jul 19 07:48:39.296617 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:39.296519 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" event={"ID":"af121060-cd61-494e-8b87-13ba67a89698","Type":"ContainerStarted","Data":"27246b3968ec643def9f245a36db479737624f452febebdaa4b615204a8bf240"} Jul 19 07:48:39.297070 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:39.296645 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:39.298177 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:39.298156 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" event={"ID":"cff2e13b-0644-4cc6-be23-f49ed171dde0","Type":"ContainerStarted","Data":"deb7acc1686eb13ba61aa44afe00685dff004ff1f07b02cccb0db25eb41c1406"} Jul 19 07:48:39.313252 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:39.313210 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" podStartSLOduration=1.596434924 podStartE2EDuration="5.313197351s" totalImagesPullingTime="3.716762427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:39.312305645 +0000 UTC m=+367.091968989" watchObservedRunningTime="2026-07-19 07:48:39.313197351 +0000 UTC m=+367.092860662" Jul 19 07:48:40.303364 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:40.303325 2574 generic.go:356] "Generic (PLEG): container finished" podID="cff2e13b-0644-4cc6-be23-f49ed171dde0" containerID="deb7acc1686eb13ba61aa44afe00685dff004ff1f07b02cccb0db25eb41c1406" exitCode=0 Jul 19 07:48:40.303801 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:40.303373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" event={"ID":"cff2e13b-0644-4cc6-be23-f49ed171dde0","Type":"ContainerDied","Data":"deb7acc1686eb13ba61aa44afe00685dff004ff1f07b02cccb0db25eb41c1406"} Jul 19 07:48:41.309400 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:41.309368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" event={"ID":"cff2e13b-0644-4cc6-be23-f49ed171dde0","Type":"ContainerStarted","Data":"2498bfdb289b72fa48e149178a3d16fc24124db3fe1d3290dac05d05681df048"} Jul 19 07:48:41.309918 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:41.309442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:41.329962 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:41.329922 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" podStartSLOduration=1.829264695 podStartE2EDuration="7.329907545s" totalImagesPullingTime="4.418607346s" totalInitContainerRuntime="1.082035504s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:41.328526942 +0000 UTC m=+369.108190247" watchObservedRunningTime="2026-07-19 07:48:41.329907545 +0000 UTC m=+369.109570861" Jul 19 07:48:50.306312 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:50.306281 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-lchls" Jul 19 07:48:52.315939 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:52.315908 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-2wlfq" Jul 19 07:48:54.754641 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.754606 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz"] Jul 19 07:48:54.757626 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.757603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:54.760062 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.760040 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 19 07:48:54.760183 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.760166 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ltrgp\"" Jul 19 07:48:54.761048 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.761020 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 19 07:48:54.761048 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.761044 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 19 07:48:54.761252 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.761069 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 19 07:48:54.768611 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.768591 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz"] Jul 19 07:48:54.912461 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.912413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tls-certs\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:54.912649 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.912490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwrl\" (UniqueName: \"kubernetes.io/projected/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-kube-api-access-fpwrl\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:54.912649 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:54.912568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tmp\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.013973 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.013862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tls-certs\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.013973 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.013918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwrl\" (UniqueName: \"kubernetes.io/projected/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-kube-api-access-fpwrl\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.013973 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.013970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tmp\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.014218 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:48:55.014035 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 19 07:48:55.014218 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:48:55.014114 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tls-certs podName:aac1fabb-3259-4c8e-8dfd-7a3d462cec9a nodeName:}" failed. No retries permitted until 2026-07-19 07:48:55.51409453 +0000 UTC m=+383.293757826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tls-certs") pod "kube-auth-proxy-56f59f876d-6h6nz" (UID: "aac1fabb-3259-4c8e-8dfd-7a3d462cec9a") : secret "kube-auth-proxy-tls" not found Jul 19 07:48:55.016357 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.016339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tmp\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.022974 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.022952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwrl\" (UniqueName: \"kubernetes.io/projected/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-kube-api-access-fpwrl\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.519702 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.519648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tls-certs\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.522229 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.522210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac1fabb-3259-4c8e-8dfd-7a3d462cec9a-tls-certs\") pod \"kube-auth-proxy-56f59f876d-6h6nz\" (UID: \"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a\") " pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.669749 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.669714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" Jul 19 07:48:55.810090 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:55.810062 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz"] Jul 19 07:48:55.812702 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:48:55.812661 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac1fabb_3259_4c8e_8dfd_7a3d462cec9a.slice/crio-7232e543d9904f9c5d0241b108d4c8316711516a9decdec733349a4f2536491b WatchSource:0}: Error finding container 7232e543d9904f9c5d0241b108d4c8316711516a9decdec733349a4f2536491b: Status 404 returned error can't find the container with id 7232e543d9904f9c5d0241b108d4c8316711516a9decdec733349a4f2536491b Jul 19 07:48:56.369763 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:56.369726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" event={"ID":"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a","Type":"ContainerStarted","Data":"7232e543d9904f9c5d0241b108d4c8316711516a9decdec733349a4f2536491b"} Jul 19 07:48:58.379168 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:58.379131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" event={"ID":"aac1fabb-3259-4c8e-8dfd-7a3d462cec9a","Type":"ContainerStarted","Data":"e0d45e03a2f235120ca0917ee4b1c5fe05da7c691839695b32f1d45edaa4c718"} Jul 19 07:48:58.394431 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:48:58.394381 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-56f59f876d-6h6nz" podStartSLOduration=2.19320681 podStartE2EDuration="4.394363941s" totalImagesPullingTime="2.201157131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:58.393061434 +0000 UTC m=+386.172724762" watchObservedRunningTime="2026-07-19 07:48:58.394363941 +0000 UTC m=+386.174027255" Jul 19 07:49:06.221311 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.221273 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kgr97"] Jul 19 07:49:06.224551 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.224534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.228037 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.228012 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bt26n\"" Jul 19 07:49:06.228292 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.228275 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 19 07:49:06.259608 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.259580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kgr97"] Jul 19 07:49:06.308435 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.308403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-cert\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.308435 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.308434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6sr5\" (UniqueName: \"kubernetes.io/projected/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-kube-api-access-t6sr5\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.409340 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.409308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-cert\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.409340 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.409341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6sr5\" (UniqueName: \"kubernetes.io/projected/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-kube-api-access-t6sr5\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.409537 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:49:06.409476 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 19 07:49:06.409574 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:49:06.409545 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-cert podName:c471b15a-d8f5-4d24-9dbf-f46b7e5f0038 nodeName:}" failed. No retries permitted until 2026-07-19 07:49:06.909524588 +0000 UTC m=+394.689187879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-cert") pod "kserve-controller-manager-d6f4b65b8-kgr97" (UID: "c471b15a-d8f5-4d24-9dbf-f46b7e5f0038") : secret "kserve-webhook-server-cert" not found Jul 19 07:49:06.419254 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.419221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6sr5\" (UniqueName: \"kubernetes.io/projected/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-kube-api-access-t6sr5\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.913792 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.913736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-cert\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:06.916303 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:06.916280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c471b15a-d8f5-4d24-9dbf-f46b7e5f0038-cert\") pod \"kserve-controller-manager-d6f4b65b8-kgr97\" (UID: \"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:07.134725 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:07.134669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:07.273799 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:07.273649 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kgr97"] Jul 19 07:49:07.276527 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:49:07.276499 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc471b15a_d8f5_4d24_9dbf_f46b7e5f0038.slice/crio-75ebda469b6d0570290c64a03aab241a05e69a69041588f0c6d590c7d5b149b1 WatchSource:0}: Error finding container 75ebda469b6d0570290c64a03aab241a05e69a69041588f0c6d590c7d5b149b1: Status 404 returned error can't find the container with id 75ebda469b6d0570290c64a03aab241a05e69a69041588f0c6d590c7d5b149b1 Jul 19 07:49:07.415549 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:07.415515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" event={"ID":"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038","Type":"ContainerStarted","Data":"75ebda469b6d0570290c64a03aab241a05e69a69041588f0c6d590c7d5b149b1"} Jul 19 07:49:10.429604 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:10.429557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" event={"ID":"c471b15a-d8f5-4d24-9dbf-f46b7e5f0038","Type":"ContainerStarted","Data":"f3e6572fb70d57d97dd3f8ab4ac18b93c665d46890a6a91e68e36a7568140adf"} Jul 19 07:49:10.430114 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:10.429725 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:49:10.445403 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:10.445351 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" podStartSLOduration=2.278290397 podStartE2EDuration="4.445333939s" totalImagesPullingTime="2.167043542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:10.444015234 +0000 UTC m=+398.223678546" watchObservedRunningTime="2026-07-19 07:49:10.445333939 +0000 UTC m=+398.224997252" Jul 19 07:49:41.439055 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:49:41.439026 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kgr97" Jul 19 07:50:42.847507 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.847475 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt"] Jul 19 07:50:42.850632 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.850616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.852790 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.852758 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 19 07:50:42.852923 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.852897 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 07:50:42.852991 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.852922 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4wmr\"" Jul 19 07:50:42.853651 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.853635 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 07:50:42.853724 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.853639 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 19 07:50:42.859551 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.859520 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt"] Jul 19 07:50:42.859655 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.859585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbr9\" (UniqueName: \"kubernetes.io/projected/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-kube-api-access-klbr9\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.859655 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.859641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.859753 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.859698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.960128 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.960085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbr9\" (UniqueName: \"kubernetes.io/projected/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-kube-api-access-klbr9\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.960128 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.960132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.960341 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.960161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.960341 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:50:42.960242 2574 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 19 07:50:42.960341 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:50:42.960306 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-plugin-serving-cert podName:63d8ba89-d8a8-4285-abcd-05672cfa1d8f nodeName:}" failed. No retries permitted until 2026-07-19 07:50:43.460288759 +0000 UTC m=+491.239952055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-4w9rt" (UID: "63d8ba89-d8a8-4285-abcd-05672cfa1d8f") : secret "plugin-serving-cert" not found Jul 19 07:50:42.960912 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.960893 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:42.968418 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:42.968389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbr9\" (UniqueName: \"kubernetes.io/projected/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-kube-api-access-klbr9\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:43.464439 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:43.464406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:43.467045 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:43.467012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63d8ba89-d8a8-4285-abcd-05672cfa1d8f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4w9rt\" (UID: \"63d8ba89-d8a8-4285-abcd-05672cfa1d8f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:43.468702 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:43.468684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" Jul 19 07:50:43.595881 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:43.595852 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt"] Jul 19 07:50:43.599142 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:50:43.599114 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63d8ba89_d8a8_4285_abcd_05672cfa1d8f.slice/crio-1169f664c36e622b731fd0e13d7d69f1c306f7549972549c3b7a84382e0e2800 WatchSource:0}: Error finding container 1169f664c36e622b731fd0e13d7d69f1c306f7549972549c3b7a84382e0e2800: Status 404 returned error can't find the container with id 1169f664c36e622b731fd0e13d7d69f1c306f7549972549c3b7a84382e0e2800 Jul 19 07:50:43.783220 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:43.783131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" event={"ID":"63d8ba89-d8a8-4285-abcd-05672cfa1d8f","Type":"ContainerStarted","Data":"1169f664c36e622b731fd0e13d7d69f1c306f7549972549c3b7a84382e0e2800"} Jul 19 07:50:52.325783 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.325748 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx"] Jul 19 07:50:52.346200 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.346168 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx"] Jul 19 07:50:52.346373 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.346284 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.350383 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.350352 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zktnv\"" Jul 19 07:50:52.440932 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.440900 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx"] Jul 19 07:50:52.441245 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:50:52.441208 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-pt8fl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" podUID="9f714fcb-2bec-4a6d-b147-0d04d9562793" Jul 19 07:50:52.451462 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.451424 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f714fcb-2bec-4a6d-b147-0d04d9562793-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.451649 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.451524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8fl\" (UniqueName: \"kubernetes.io/projected/9f714fcb-2bec-4a6d-b147-0d04d9562793-kube-api-access-pt8fl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.552426 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.552383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8fl\" (UniqueName: \"kubernetes.io/projected/9f714fcb-2bec-4a6d-b147-0d04d9562793-kube-api-access-pt8fl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.552613 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.552507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f714fcb-2bec-4a6d-b147-0d04d9562793-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.552880 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.552857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f714fcb-2bec-4a6d-b147-0d04d9562793-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.582401 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.582321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8fl\" (UniqueName: \"kubernetes.io/projected/9f714fcb-2bec-4a6d-b147-0d04d9562793-kube-api-access-pt8fl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.829696 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.829643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.846699 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.846658 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:52.885908 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.885870 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx"] Jul 19 07:50:52.892738 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.892708 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx"] Jul 19 07:50:52.931067 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.930999 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp"] Jul 19 07:50:52.936067 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.936038 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:52.948332 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.948299 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp"] Jul 19 07:50:52.956960 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.956929 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt8fl\" (UniqueName: \"kubernetes.io/projected/9f714fcb-2bec-4a6d-b147-0d04d9562793-kube-api-access-pt8fl\") pod \"9f714fcb-2bec-4a6d-b147-0d04d9562793\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " Jul 19 07:50:52.957107 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.957065 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f714fcb-2bec-4a6d-b147-0d04d9562793-extensions-socket-volume\") pod \"9f714fcb-2bec-4a6d-b147-0d04d9562793\" (UID: \"9f714fcb-2bec-4a6d-b147-0d04d9562793\") " Jul 19 07:50:52.957372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.957341 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f714fcb-2bec-4a6d-b147-0d04d9562793-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9f714fcb-2bec-4a6d-b147-0d04d9562793" (UID: "9f714fcb-2bec-4a6d-b147-0d04d9562793"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:50:52.959801 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:52.959752 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f714fcb-2bec-4a6d-b147-0d04d9562793-kube-api-access-pt8fl" (OuterVolumeSpecName: "kube-api-access-pt8fl") pod "9f714fcb-2bec-4a6d-b147-0d04d9562793" (UID: "9f714fcb-2bec-4a6d-b147-0d04d9562793"). InnerVolumeSpecName "kube-api-access-pt8fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:50:53.058177 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.058136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3592903a-b6f4-482b-983e-a0291447b982-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rc2zp\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.058371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.058271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clr84\" (UniqueName: \"kubernetes.io/projected/3592903a-b6f4-482b-983e-a0291447b982-kube-api-access-clr84\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rc2zp\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.058371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.058329 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f714fcb-2bec-4a6d-b147-0d04d9562793-extensions-socket-volume\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.058371 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.058339 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt8fl\" (UniqueName: \"kubernetes.io/projected/9f714fcb-2bec-4a6d-b147-0d04d9562793-kube-api-access-pt8fl\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:50:53.159781 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.159656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clr84\" (UniqueName: \"kubernetes.io/projected/3592903a-b6f4-482b-983e-a0291447b982-kube-api-access-clr84\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rc2zp\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.159960 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.159807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3592903a-b6f4-482b-983e-a0291447b982-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rc2zp\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.160265 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.160238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3592903a-b6f4-482b-983e-a0291447b982-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rc2zp\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.168017 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.167989 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clr84\" (UniqueName: \"kubernetes.io/projected/3592903a-b6f4-482b-983e-a0291447b982-kube-api-access-clr84\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rc2zp\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.250963 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.250924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:50:53.834664 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:53.834630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:53.837178 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:50:53.837146 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" is forbidden: User \"system:node:ip-10-0-132-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-126.ec2.internal' and this object" podUID="9f714fcb-2bec-4a6d-b147-0d04d9562793" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:53.846538 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:50:53.846498 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-qqndx\" is forbidden: User \"system:node:ip-10-0-132-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-126.ec2.internal' and this object" podUID="9f714fcb-2bec-4a6d-b147-0d04d9562793" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qqndx" Jul 19 07:50:54.732175 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:50:54.732145 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f714fcb-2bec-4a6d-b147-0d04d9562793" path="/var/lib/kubelet/pods/9f714fcb-2bec-4a6d-b147-0d04d9562793/volumes" Jul 19 07:51:04.518094 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:04.518066 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp"] Jul 19 07:51:04.519543 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:51:04.519505 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3592903a_b6f4_482b_983e_a0291447b982.slice/crio-b2e6122a8fd0b7c10877e34a71fa3bdd45d053e4b4fc72885d24c3738b67122d WatchSource:0}: Error finding container b2e6122a8fd0b7c10877e34a71fa3bdd45d053e4b4fc72885d24c3738b67122d: Status 404 returned error can't find the container with id b2e6122a8fd0b7c10877e34a71fa3bdd45d053e4b4fc72885d24c3738b67122d Jul 19 07:51:04.889704 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:04.889639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" event={"ID":"63d8ba89-d8a8-4285-abcd-05672cfa1d8f","Type":"ContainerStarted","Data":"7b695d50d24d8570e1be0d322acdf65e540f2103ae2a327e6653c55171124b62"} Jul 19 07:51:04.891044 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:04.891015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" event={"ID":"3592903a-b6f4-482b-983e-a0291447b982","Type":"ContainerStarted","Data":"b2e6122a8fd0b7c10877e34a71fa3bdd45d053e4b4fc72885d24c3738b67122d"} Jul 19 07:51:04.904995 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:04.904943 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4w9rt" podStartSLOduration=2.06785136 podStartE2EDuration="22.904921501s" totalImagesPullingTime="20.837070141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:04.904055991 +0000 UTC m=+512.683719306" watchObservedRunningTime="2026-07-19 07:51:04.904921501 +0000 UTC m=+512.684584816" Jul 19 07:51:05.248047 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:05.247970 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp"] Jul 19 07:51:09.916071 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:09.916036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" event={"ID":"3592903a-b6f4-482b-983e-a0291447b982","Type":"ContainerStarted","Data":"1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee"} Jul 19 07:51:09.916542 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:09.916072 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" podUID="3592903a-b6f4-482b-983e-a0291447b982" containerName="manager" containerID="cri-o://1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee" gracePeriod=10 Jul 19 07:51:09.916542 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:09.916152 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:51:09.933660 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:09.933606 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" podStartSLOduration=13.414992282 podStartE2EDuration="17.933586582s" totalImagesPullingTime="4.5185943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:09.932456311 +0000 UTC m=+517.712119646" watchObservedRunningTime="2026-07-19 07:51:09.933586582 +0000 UTC m=+517.713249898" Jul 19 07:51:10.732336 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.732311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:51:10.831619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.831542 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3592903a-b6f4-482b-983e-a0291447b982-extensions-socket-volume\") pod \"3592903a-b6f4-482b-983e-a0291447b982\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " Jul 19 07:51:10.831619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.831576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clr84\" (UniqueName: \"kubernetes.io/projected/3592903a-b6f4-482b-983e-a0291447b982-kube-api-access-clr84\") pod \"3592903a-b6f4-482b-983e-a0291447b982\" (UID: \"3592903a-b6f4-482b-983e-a0291447b982\") " Jul 19 07:51:10.832196 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.832118 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3592903a-b6f4-482b-983e-a0291447b982-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3592903a-b6f4-482b-983e-a0291447b982" (UID: "3592903a-b6f4-482b-983e-a0291447b982"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 07:51:10.834182 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.834159 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3592903a-b6f4-482b-983e-a0291447b982-kube-api-access-clr84" (OuterVolumeSpecName: "kube-api-access-clr84") pod "3592903a-b6f4-482b-983e-a0291447b982" (UID: "3592903a-b6f4-482b-983e-a0291447b982"). InnerVolumeSpecName "kube-api-access-clr84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:51:10.921030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.920995 2574 generic.go:356] "Generic (PLEG): container finished" podID="3592903a-b6f4-482b-983e-a0291447b982" containerID="1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee" exitCode=0 Jul 19 07:51:10.921469 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.921042 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" Jul 19 07:51:10.921469 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.921041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" event={"ID":"3592903a-b6f4-482b-983e-a0291447b982","Type":"ContainerDied","Data":"1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee"} Jul 19 07:51:10.921469 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.921187 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp" event={"ID":"3592903a-b6f4-482b-983e-a0291447b982","Type":"ContainerDied","Data":"b2e6122a8fd0b7c10877e34a71fa3bdd45d053e4b4fc72885d24c3738b67122d"} Jul 19 07:51:10.921469 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.921217 2574 scope.go:117] "RemoveContainer" containerID="1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee" Jul 19 07:51:10.930353 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.930337 2574 scope.go:117] "RemoveContainer" containerID="1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee" Jul 19 07:51:10.930621 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:51:10.930601 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee\": container with ID starting with 1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee not found: ID does not exist" containerID="1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee" Jul 19 07:51:10.930671 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.930629 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee"} err="failed to get container status \"1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee\": rpc error: code = NotFound desc = could not find container \"1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee\": container with ID starting with 1aadb20b34fa82d2a48db648ef48a61d8a7a25b8fd2308c2021232fb4a72bfee not found: ID does not exist" Jul 19 07:51:10.932607 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.932587 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3592903a-b6f4-482b-983e-a0291447b982-extensions-socket-volume\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:51:10.932607 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.932607 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clr84\" (UniqueName: \"kubernetes.io/projected/3592903a-b6f4-482b-983e-a0291447b982-kube-api-access-clr84\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:51:10.941275 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.941249 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp"] Jul 19 07:51:10.947067 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:10.947047 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rc2zp"] Jul 19 07:51:12.731483 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:12.731446 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3592903a-b6f4-482b-983e-a0291447b982" path="/var/lib/kubelet/pods/3592903a-b6f4-482b-983e-a0291447b982/volumes" Jul 19 07:51:38.058701 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.058650 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:38.059123 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.058986 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3592903a-b6f4-482b-983e-a0291447b982" containerName="manager" Jul 19 07:51:38.059123 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.058997 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3592903a-b6f4-482b-983e-a0291447b982" containerName="manager" Jul 19 07:51:38.059123 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.059068 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3592903a-b6f4-482b-983e-a0291447b982" containerName="manager" Jul 19 07:51:38.095698 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.095641 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:38.095886 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.095718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.098334 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.098310 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 19 07:51:38.155970 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.155933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrzt\" (UniqueName: \"kubernetes.io/projected/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-kube-api-access-5mrzt\") pod \"limitador-limitador-64c94fd64f-hqlwz\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.156133 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.156051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-config-file\") pod \"limitador-limitador-64c94fd64f-hqlwz\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.156588 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.156571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:38.256962 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.256928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-config-file\") pod \"limitador-limitador-64c94fd64f-hqlwz\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.257139 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.257006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrzt\" (UniqueName: \"kubernetes.io/projected/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-kube-api-access-5mrzt\") pod \"limitador-limitador-64c94fd64f-hqlwz\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.257573 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.257552 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-config-file\") pod \"limitador-limitador-64c94fd64f-hqlwz\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.264205 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.264176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrzt\" (UniqueName: \"kubernetes.io/projected/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-kube-api-access-5mrzt\") pod \"limitador-limitador-64c94fd64f-hqlwz\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.405321 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.405289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:38.538872 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.538840 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:38.541584 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:51:38.541556 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb7cb85_0aa9_40f8_a837_007f89f4ffe6.slice/crio-822bfff76b973b63cd0ec9dfab6b569b72fa4384508cb309fdea7e66564c279f WatchSource:0}: Error finding container 822bfff76b973b63cd0ec9dfab6b569b72fa4384508cb309fdea7e66564c279f: Status 404 returned error can't find the container with id 822bfff76b973b63cd0ec9dfab6b569b72fa4384508cb309fdea7e66564c279f Jul 19 07:51:38.858777 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.858736 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-7wxbn"] Jul 19 07:51:38.863404 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.863377 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:38.865619 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.865597 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bqx2c\"" Jul 19 07:51:38.868751 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.868724 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7wxbn"] Jul 19 07:51:38.961044 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:38.961005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2b2\" (UniqueName: \"kubernetes.io/projected/1e5cb85c-fb87-4975-b1c3-997275dc2103-kube-api-access-ws2b2\") pod \"authorino-58cd955675-7wxbn\" (UID: \"1e5cb85c-fb87-4975-b1c3-997275dc2103\") " pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:39.049131 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:39.049100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" event={"ID":"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6","Type":"ContainerStarted","Data":"822bfff76b973b63cd0ec9dfab6b569b72fa4384508cb309fdea7e66564c279f"} Jul 19 07:51:39.061567 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:39.061535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2b2\" (UniqueName: \"kubernetes.io/projected/1e5cb85c-fb87-4975-b1c3-997275dc2103-kube-api-access-ws2b2\") pod \"authorino-58cd955675-7wxbn\" (UID: \"1e5cb85c-fb87-4975-b1c3-997275dc2103\") " pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:39.072658 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:39.072619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2b2\" (UniqueName: \"kubernetes.io/projected/1e5cb85c-fb87-4975-b1c3-997275dc2103-kube-api-access-ws2b2\") pod \"authorino-58cd955675-7wxbn\" (UID: \"1e5cb85c-fb87-4975-b1c3-997275dc2103\") " pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:39.174129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:39.174041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:39.358575 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:39.358536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7wxbn"] Jul 19 07:51:39.363930 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:51:39.363899 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5cb85c_fb87_4975_b1c3_997275dc2103.slice/crio-4073c555e2046a13299891dfac8ff6aafa7a1253809bbc1b6f418d22b88c4931 WatchSource:0}: Error finding container 4073c555e2046a13299891dfac8ff6aafa7a1253809bbc1b6f418d22b88c4931: Status 404 returned error can't find the container with id 4073c555e2046a13299891dfac8ff6aafa7a1253809bbc1b6f418d22b88c4931 Jul 19 07:51:40.054869 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:40.054825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7wxbn" event={"ID":"1e5cb85c-fb87-4975-b1c3-997275dc2103","Type":"ContainerStarted","Data":"4073c555e2046a13299891dfac8ff6aafa7a1253809bbc1b6f418d22b88c4931"} Jul 19 07:51:42.071501 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:42.071460 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" event={"ID":"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6","Type":"ContainerStarted","Data":"da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788"} Jul 19 07:51:42.071963 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:42.071570 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:42.088166 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:42.088111 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" podStartSLOduration=1.183277804 podStartE2EDuration="4.088093834s" totalImagesPullingTime="2.90481603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:42.086258649 +0000 UTC m=+549.865921967" watchObservedRunningTime="2026-07-19 07:51:42.088093834 +0000 UTC m=+549.867757148" Jul 19 07:51:44.081728 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:44.081611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7wxbn" event={"ID":"1e5cb85c-fb87-4975-b1c3-997275dc2103","Type":"ContainerStarted","Data":"0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a"} Jul 19 07:51:44.095117 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:44.095064 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-7wxbn" podStartSLOduration=1.7952073309999999 podStartE2EDuration="6.095041556s" totalImagesPullingTime="4.299834225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:44.094350707 +0000 UTC m=+551.874014022" watchObservedRunningTime="2026-07-19 07:51:44.095041556 +0000 UTC m=+551.874704870" Jul 19 07:51:44.495700 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:44.495646 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7wxbn"] Jul 19 07:51:46.089482 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:46.089441 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-7wxbn" podUID="1e5cb85c-fb87-4975-b1c3-997275dc2103" containerName="authorino" containerID="cri-o://0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a" gracePeriod=30 Jul 19 07:51:46.342012 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:46.341949 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:46.433413 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:46.433362 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws2b2\" (UniqueName: \"kubernetes.io/projected/1e5cb85c-fb87-4975-b1c3-997275dc2103-kube-api-access-ws2b2\") pod \"1e5cb85c-fb87-4975-b1c3-997275dc2103\" (UID: \"1e5cb85c-fb87-4975-b1c3-997275dc2103\") " Jul 19 07:51:46.435601 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:46.435569 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5cb85c-fb87-4975-b1c3-997275dc2103-kube-api-access-ws2b2" (OuterVolumeSpecName: "kube-api-access-ws2b2") pod "1e5cb85c-fb87-4975-b1c3-997275dc2103" (UID: "1e5cb85c-fb87-4975-b1c3-997275dc2103"). InnerVolumeSpecName "kube-api-access-ws2b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:51:46.534497 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:46.534458 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2b2\" (UniqueName: \"kubernetes.io/projected/1e5cb85c-fb87-4975-b1c3-997275dc2103-kube-api-access-ws2b2\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:51:47.094522 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.094488 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e5cb85c-fb87-4975-b1c3-997275dc2103" containerID="0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a" exitCode=0 Jul 19 07:51:47.095107 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.094535 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7wxbn" Jul 19 07:51:47.095107 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.094581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7wxbn" event={"ID":"1e5cb85c-fb87-4975-b1c3-997275dc2103","Type":"ContainerDied","Data":"0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a"} Jul 19 07:51:47.095107 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.094621 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7wxbn" event={"ID":"1e5cb85c-fb87-4975-b1c3-997275dc2103","Type":"ContainerDied","Data":"4073c555e2046a13299891dfac8ff6aafa7a1253809bbc1b6f418d22b88c4931"} Jul 19 07:51:47.095107 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.094638 2574 scope.go:117] "RemoveContainer" containerID="0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a" Jul 19 07:51:47.103012 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.102996 2574 scope.go:117] "RemoveContainer" containerID="0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a" Jul 19 07:51:47.103278 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:51:47.103251 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a\": container with ID starting with 0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a not found: ID does not exist" containerID="0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a" Jul 19 07:51:47.103378 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.103277 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a"} err="failed to get container status \"0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a\": rpc error: code = NotFound desc = could not find container \"0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a\": container with ID starting with 0188f633d3c4538f14c30a871f7f7078dadd83147e3d6b640bb6360e15f2727a not found: ID does not exist" Jul 19 07:51:47.111248 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.111223 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7wxbn"] Jul 19 07:51:47.112841 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:47.112822 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-7wxbn"] Jul 19 07:51:48.731515 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:48.731483 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e5cb85c-fb87-4975-b1c3-997275dc2103" path="/var/lib/kubelet/pods/1e5cb85c-fb87-4975-b1c3-997275dc2103/volumes" Jul 19 07:51:53.077522 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:53.077493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:55.506655 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:55.506623 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:55.507075 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:55.506856 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" podUID="8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" containerName="limitador" containerID="cri-o://da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788" gracePeriod=30 Jul 19 07:51:56.059614 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.059592 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:56.136574 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.136544 2574 generic.go:356] "Generic (PLEG): container finished" podID="8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" containerID="da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788" exitCode=0 Jul 19 07:51:56.136789 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.136588 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" Jul 19 07:51:56.136789 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.136601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" event={"ID":"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6","Type":"ContainerDied","Data":"da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788"} Jul 19 07:51:56.136789 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.136643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hqlwz" event={"ID":"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6","Type":"ContainerDied","Data":"822bfff76b973b63cd0ec9dfab6b569b72fa4384508cb309fdea7e66564c279f"} Jul 19 07:51:56.136789 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.136665 2574 scope.go:117] "RemoveContainer" containerID="da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788" Jul 19 07:51:56.145030 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.145014 2574 scope.go:117] "RemoveContainer" containerID="da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788" Jul 19 07:51:56.145274 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:51:56.145257 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788\": container with ID starting with da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788 not found: ID does not exist" containerID="da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788" Jul 19 07:51:56.145316 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.145289 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788"} err="failed to get container status \"da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788\": rpc error: code = NotFound desc = could not find container \"da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788\": container with ID starting with da45798ef9eaccd5bd07eec6e37ab3ec3117a4c32a2c092943b78ca37daeb788 not found: ID does not exist" Jul 19 07:51:56.220113 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.220084 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-config-file\") pod \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " Jul 19 07:51:56.220277 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.220184 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mrzt\" (UniqueName: \"kubernetes.io/projected/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-kube-api-access-5mrzt\") pod \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\" (UID: \"8eb7cb85-0aa9-40f8-a837-007f89f4ffe6\") " Jul 19 07:51:56.220438 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.220411 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-config-file" (OuterVolumeSpecName: "config-file") pod "8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" (UID: "8eb7cb85-0aa9-40f8-a837-007f89f4ffe6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 07:51:56.222335 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.222304 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-kube-api-access-5mrzt" (OuterVolumeSpecName: "kube-api-access-5mrzt") pod "8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" (UID: "8eb7cb85-0aa9-40f8-a837-007f89f4ffe6"). InnerVolumeSpecName "kube-api-access-5mrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:51:56.321698 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.321634 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-config-file\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:51:56.321698 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.321665 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrzt\" (UniqueName: \"kubernetes.io/projected/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6-kube-api-access-5mrzt\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:51:56.456757 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.456725 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:56.457932 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.457911 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hqlwz"] Jul 19 07:51:56.731744 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:51:56.731649 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" path="/var/lib/kubelet/pods/8eb7cb85-0aa9-40f8-a837-007f89f4ffe6/volumes" Jul 19 07:52:00.939211 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939178 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-49dbs"] Jul 19 07:52:00.939579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939485 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5cb85c-fb87-4975-b1c3-997275dc2103" containerName="authorino" Jul 19 07:52:00.939579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939496 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5cb85c-fb87-4975-b1c3-997275dc2103" containerName="authorino" Jul 19 07:52:00.939579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939513 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" containerName="limitador" Jul 19 07:52:00.939579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939518 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" containerName="limitador" Jul 19 07:52:00.939579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939580 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb7cb85-0aa9-40f8-a837-007f89f4ffe6" containerName="limitador" Jul 19 07:52:00.939761 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.939589 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5cb85c-fb87-4975-b1c3-997275dc2103" containerName="authorino" Jul 19 07:52:00.943430 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.943400 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:00.945928 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.945904 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 07:52:00.946623 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.946600 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 19 07:52:00.946623 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.946622 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2v4bk\"" Jul 19 07:52:00.946853 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.946623 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 07:52:00.950596 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:00.950574 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-49dbs"] Jul 19 07:52:01.065472 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.065441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/518d77d6-702e-4aca-ae86-fd2fae906ab6-data\") pod \"postgres-69fb4594cf-49dbs\" (UID: \"518d77d6-702e-4aca-ae86-fd2fae906ab6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.065660 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.065513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jspb\" (UniqueName: \"kubernetes.io/projected/518d77d6-702e-4aca-ae86-fd2fae906ab6-kube-api-access-9jspb\") pod \"postgres-69fb4594cf-49dbs\" (UID: \"518d77d6-702e-4aca-ae86-fd2fae906ab6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.166519 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.166479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jspb\" (UniqueName: \"kubernetes.io/projected/518d77d6-702e-4aca-ae86-fd2fae906ab6-kube-api-access-9jspb\") pod \"postgres-69fb4594cf-49dbs\" (UID: \"518d77d6-702e-4aca-ae86-fd2fae906ab6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.166756 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.166541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/518d77d6-702e-4aca-ae86-fd2fae906ab6-data\") pod \"postgres-69fb4594cf-49dbs\" (UID: \"518d77d6-702e-4aca-ae86-fd2fae906ab6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.166968 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.166947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/518d77d6-702e-4aca-ae86-fd2fae906ab6-data\") pod \"postgres-69fb4594cf-49dbs\" (UID: \"518d77d6-702e-4aca-ae86-fd2fae906ab6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.176761 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.176737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jspb\" (UniqueName: \"kubernetes.io/projected/518d77d6-702e-4aca-ae86-fd2fae906ab6-kube-api-access-9jspb\") pod \"postgres-69fb4594cf-49dbs\" (UID: \"518d77d6-702e-4aca-ae86-fd2fae906ab6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.256140 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.256053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:01.383381 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:01.383349 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-49dbs"] Jul 19 07:52:01.386157 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:52:01.386129 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518d77d6_702e_4aca_ae86_fd2fae906ab6.slice/crio-f601baef24f1c762bf5d1825c64aa0ee3fdbbd373080ec4e8f95c8f2df97a70b WatchSource:0}: Error finding container f601baef24f1c762bf5d1825c64aa0ee3fdbbd373080ec4e8f95c8f2df97a70b: Status 404 returned error can't find the container with id f601baef24f1c762bf5d1825c64aa0ee3fdbbd373080ec4e8f95c8f2df97a70b Jul 19 07:52:02.165816 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:02.165776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" event={"ID":"518d77d6-702e-4aca-ae86-fd2fae906ab6","Type":"ContainerStarted","Data":"f601baef24f1c762bf5d1825c64aa0ee3fdbbd373080ec4e8f95c8f2df97a70b"} Jul 19 07:52:06.188197 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:06.188158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" event={"ID":"518d77d6-702e-4aca-ae86-fd2fae906ab6","Type":"ContainerStarted","Data":"e7ccc71a8fdd982e602c3fa7fa6ccd0a02824db19bbe7878097fd8aedfc23be4"} Jul 19 07:52:06.188579 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:06.188312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:06.214396 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:06.214331 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" podStartSLOduration=1.593560836 podStartE2EDuration="6.214306933s" totalImagesPullingTime="4.620746097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:06.211946523 +0000 UTC m=+573.991609853" watchObservedRunningTime="2026-07-19 07:52:06.214306933 +0000 UTC m=+573.993970248" Jul 19 07:52:12.222977 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:12.222943 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-49dbs" Jul 19 07:52:19.271383 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.271353 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86bdfc85f9-z7ksz"] Jul 19 07:52:19.479640 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.479604 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bdfc85f9-z7ksz"] Jul 19 07:52:19.479640 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.479648 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f4f5887c-xktn7"] Jul 19 07:52:19.479855 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.479745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.483104 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.483082 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 19 07:52:19.483104 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.483089 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 19 07:52:19.483312 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.483080 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w8hzw\"" Jul 19 07:52:19.510026 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.510000 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-xktn7"] Jul 19 07:52:19.510154 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.510079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.518964 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.518938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d848cd-c1e8-40be-b518-af95c384bf2f-webhook-cert\") pod \"maas-controller-86bdfc85f9-z7ksz\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.519093 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.518967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7j2\" (UniqueName: \"kubernetes.io/projected/e112e106-3b10-485f-9adc-9740bae77219-kube-api-access-9h7j2\") pod \"maas-controller-58f4f5887c-xktn7\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.519093 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.519056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmwk\" (UniqueName: \"kubernetes.io/projected/60d848cd-c1e8-40be-b518-af95c384bf2f-kube-api-access-bqmwk\") pod \"maas-controller-86bdfc85f9-z7ksz\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.519193 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.519166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e112e106-3b10-485f-9adc-9740bae77219-webhook-cert\") pod \"maas-controller-58f4f5887c-xktn7\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.620390 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.620361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e112e106-3b10-485f-9adc-9740bae77219-webhook-cert\") pod \"maas-controller-58f4f5887c-xktn7\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.620390 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.620400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d848cd-c1e8-40be-b518-af95c384bf2f-webhook-cert\") pod \"maas-controller-86bdfc85f9-z7ksz\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.620671 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.620428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7j2\" (UniqueName: \"kubernetes.io/projected/e112e106-3b10-485f-9adc-9740bae77219-kube-api-access-9h7j2\") pod \"maas-controller-58f4f5887c-xktn7\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.620671 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.620504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmwk\" (UniqueName: \"kubernetes.io/projected/60d848cd-c1e8-40be-b518-af95c384bf2f-kube-api-access-bqmwk\") pod \"maas-controller-86bdfc85f9-z7ksz\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.623169 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.623137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e112e106-3b10-485f-9adc-9740bae77219-webhook-cert\") pod \"maas-controller-58f4f5887c-xktn7\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.623281 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.623137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d848cd-c1e8-40be-b518-af95c384bf2f-webhook-cert\") pod \"maas-controller-86bdfc85f9-z7ksz\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.627938 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.627912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmwk\" (UniqueName: \"kubernetes.io/projected/60d848cd-c1e8-40be-b518-af95c384bf2f-kube-api-access-bqmwk\") pod \"maas-controller-86bdfc85f9-z7ksz\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.628257 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.628235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7j2\" (UniqueName: \"kubernetes.io/projected/e112e106-3b10-485f-9adc-9740bae77219-kube-api-access-9h7j2\") pod \"maas-controller-58f4f5887c-xktn7\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.789661 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.789631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:19.819819 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.819786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:19.975657 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:19.975626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bdfc85f9-z7ksz"] Jul 19 07:52:19.977769 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:52:19.977739 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d848cd_c1e8_40be_b518_af95c384bf2f.slice/crio-d86f96a2b1c71de32536c3917026df1c153dcda541759003ffac02fc70803dcb WatchSource:0}: Error finding container d86f96a2b1c71de32536c3917026df1c153dcda541759003ffac02fc70803dcb: Status 404 returned error can't find the container with id d86f96a2b1c71de32536c3917026df1c153dcda541759003ffac02fc70803dcb Jul 19 07:52:20.003654 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:20.003632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-xktn7"] Jul 19 07:52:20.005541 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:52:20.005517 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode112e106_3b10_485f_9adc_9740bae77219.slice/crio-ba7ce2c4d956dfa83493deab8d08b06de316a515d9ffe55a66de94b6a8b56f34 WatchSource:0}: Error finding container ba7ce2c4d956dfa83493deab8d08b06de316a515d9ffe55a66de94b6a8b56f34: Status 404 returned error can't find the container with id ba7ce2c4d956dfa83493deab8d08b06de316a515d9ffe55a66de94b6a8b56f34 Jul 19 07:52:20.249072 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:20.248977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-xktn7" event={"ID":"e112e106-3b10-485f-9adc-9740bae77219","Type":"ContainerStarted","Data":"ba7ce2c4d956dfa83493deab8d08b06de316a515d9ffe55a66de94b6a8b56f34"} Jul 19 07:52:20.250086 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:20.250063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" event={"ID":"60d848cd-c1e8-40be-b518-af95c384bf2f","Type":"ContainerStarted","Data":"d86f96a2b1c71de32536c3917026df1c153dcda541759003ffac02fc70803dcb"} Jul 19 07:52:23.268736 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:23.268667 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-xktn7" event={"ID":"e112e106-3b10-485f-9adc-9740bae77219","Type":"ContainerStarted","Data":"237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683"} Jul 19 07:52:23.270510 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:23.270470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" event={"ID":"60d848cd-c1e8-40be-b518-af95c384bf2f","Type":"ContainerStarted","Data":"08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5"} Jul 19 07:52:23.270660 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:23.270605 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:23.289454 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:23.289406 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f4f5887c-xktn7" podStartSLOduration=1.675642201 podStartE2EDuration="4.289390108s" totalImagesPullingTime="2.613747907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:23.288203491 +0000 UTC m=+591.067866832" watchObservedRunningTime="2026-07-19 07:52:23.289390108 +0000 UTC m=+591.069053422" Jul 19 07:52:23.305007 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:23.304952 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" podStartSLOduration=1.654460449 podStartE2EDuration="4.304935655s" totalImagesPullingTime="2.650475206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:23.304067921 +0000 UTC m=+591.083731248" watchObservedRunningTime="2026-07-19 07:52:23.304935655 +0000 UTC m=+591.084598968" Jul 19 07:52:24.275747 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:24.275716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:34.277635 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:34.277603 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:35.282089 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.282058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:52:35.320441 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.320411 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bdfc85f9-z7ksz"] Jul 19 07:52:35.320644 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.320600 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" podUID="60d848cd-c1e8-40be-b518-af95c384bf2f" containerName="manager" containerID="cri-o://08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5" gracePeriod=30 Jul 19 07:52:35.574614 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.574591 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:35.672956 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.672919 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqmwk\" (UniqueName: \"kubernetes.io/projected/60d848cd-c1e8-40be-b518-af95c384bf2f-kube-api-access-bqmwk\") pod \"60d848cd-c1e8-40be-b518-af95c384bf2f\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " Jul 19 07:52:35.673129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.672987 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d848cd-c1e8-40be-b518-af95c384bf2f-webhook-cert\") pod \"60d848cd-c1e8-40be-b518-af95c384bf2f\" (UID: \"60d848cd-c1e8-40be-b518-af95c384bf2f\") " Jul 19 07:52:35.675422 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.675390 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d848cd-c1e8-40be-b518-af95c384bf2f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "60d848cd-c1e8-40be-b518-af95c384bf2f" (UID: "60d848cd-c1e8-40be-b518-af95c384bf2f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:52:35.675422 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.675397 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d848cd-c1e8-40be-b518-af95c384bf2f-kube-api-access-bqmwk" (OuterVolumeSpecName: "kube-api-access-bqmwk") pod "60d848cd-c1e8-40be-b518-af95c384bf2f" (UID: "60d848cd-c1e8-40be-b518-af95c384bf2f"). InnerVolumeSpecName "kube-api-access-bqmwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:52:35.773805 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.773775 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmwk\" (UniqueName: \"kubernetes.io/projected/60d848cd-c1e8-40be-b518-af95c384bf2f-kube-api-access-bqmwk\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:52:35.773805 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:35.773801 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d848cd-c1e8-40be-b518-af95c384bf2f-webhook-cert\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:52:36.330694 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.330645 2574 generic.go:356] "Generic (PLEG): container finished" podID="60d848cd-c1e8-40be-b518-af95c384bf2f" containerID="08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5" exitCode=0 Jul 19 07:52:36.331205 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.330709 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" Jul 19 07:52:36.331205 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.330727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" event={"ID":"60d848cd-c1e8-40be-b518-af95c384bf2f","Type":"ContainerDied","Data":"08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5"} Jul 19 07:52:36.331205 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.330786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bdfc85f9-z7ksz" event={"ID":"60d848cd-c1e8-40be-b518-af95c384bf2f","Type":"ContainerDied","Data":"d86f96a2b1c71de32536c3917026df1c153dcda541759003ffac02fc70803dcb"} Jul 19 07:52:36.331205 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.330820 2574 scope.go:117] "RemoveContainer" containerID="08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5" Jul 19 07:52:36.339637 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.339615 2574 scope.go:117] "RemoveContainer" containerID="08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5" Jul 19 07:52:36.339925 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:52:36.339911 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5\": container with ID starting with 08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5 not found: ID does not exist" containerID="08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5" Jul 19 07:52:36.339989 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.339935 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5"} err="failed to get container status \"08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5\": rpc error: code = NotFound desc = could not find container \"08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5\": container with ID starting with 08bdc5fe0ddd49ad4a282d8b984fa545608f25f8d720d8a76ec2513633f2c4e5 not found: ID does not exist" Jul 19 07:52:36.351845 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.351812 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bdfc85f9-z7ksz"] Jul 19 07:52:36.354038 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.354016 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86bdfc85f9-z7ksz"] Jul 19 07:52:36.732232 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:36.732200 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d848cd-c1e8-40be-b518-af95c384bf2f" path="/var/lib/kubelet/pods/60d848cd-c1e8-40be-b518-af95c384bf2f/volumes" Jul 19 07:52:55.235297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.235213 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c"] Jul 19 07:52:55.235747 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.235576 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d848cd-c1e8-40be-b518-af95c384bf2f" containerName="manager" Jul 19 07:52:55.235747 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.235587 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d848cd-c1e8-40be-b518-af95c384bf2f" containerName="manager" Jul 19 07:52:55.235747 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.235635 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d848cd-c1e8-40be-b518-af95c384bf2f" containerName="manager" Jul 19 07:52:55.238536 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.238519 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.240813 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.240789 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 19 07:52:55.240952 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.240926 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tdbkh\"" Jul 19 07:52:55.248481 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.248459 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c"] Jul 19 07:52:55.347833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.347802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dcbaf9-aedd-4331-a0ef-0f247158d051-openshift-service-ca\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.347833 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.347838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95dcbaf9-aedd-4331-a0ef-0f247158d051-maas-api-tls\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.348044 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.347917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2kn\" (UniqueName: \"kubernetes.io/projected/95dcbaf9-aedd-4331-a0ef-0f247158d051-kube-api-access-pm2kn\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.449140 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.449108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2kn\" (UniqueName: \"kubernetes.io/projected/95dcbaf9-aedd-4331-a0ef-0f247158d051-kube-api-access-pm2kn\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.449322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.449198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dcbaf9-aedd-4331-a0ef-0f247158d051-openshift-service-ca\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.449322 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.449227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95dcbaf9-aedd-4331-a0ef-0f247158d051-maas-api-tls\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.449983 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.449954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95dcbaf9-aedd-4331-a0ef-0f247158d051-openshift-service-ca\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.451801 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.451779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95dcbaf9-aedd-4331-a0ef-0f247158d051-maas-api-tls\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.459748 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.459728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2kn\" (UniqueName: \"kubernetes.io/projected/95dcbaf9-aedd-4331-a0ef-0f247158d051-kube-api-access-pm2kn\") pod \"maas-api-8694597d9b-n8b7c\" (UID: \"95dcbaf9-aedd-4331-a0ef-0f247158d051\") " pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.549927 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.549844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:55.690241 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.690141 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c"] Jul 19 07:52:55.692500 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:52:55.692473 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dcbaf9_aedd_4331_a0ef_0f247158d051.slice/crio-114d476f221143326bc04fb8be764e3ac68391bb4ca4eb500fca831230820f86 WatchSource:0}: Error finding container 114d476f221143326bc04fb8be764e3ac68391bb4ca4eb500fca831230820f86: Status 404 returned error can't find the container with id 114d476f221143326bc04fb8be764e3ac68391bb4ca4eb500fca831230820f86 Jul 19 07:52:55.835129 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.835100 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-q7f2k"] Jul 19 07:52:55.839612 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.839588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:55.842055 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.842030 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 19 07:52:55.842165 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.842030 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-827ps\"" Jul 19 07:52:55.845902 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.845875 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-q7f2k"] Jul 19 07:52:55.953761 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.953732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxrh\" (UniqueName: \"kubernetes.io/projected/4550ca18-d965-4ccb-a38e-c46d0dc90aaa-kube-api-access-6vxrh\") pod \"payload-processing-5bf47cd988-q7f2k\" (UID: \"4550ca18-d965-4ccb-a38e-c46d0dc90aaa\") " pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:55.953939 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:55.953815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4550ca18-d965-4ccb-a38e-c46d0dc90aaa-plugins-config-volume\") pod \"payload-processing-5bf47cd988-q7f2k\" (UID: \"4550ca18-d965-4ccb-a38e-c46d0dc90aaa\") " pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:56.054767 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.054731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4550ca18-d965-4ccb-a38e-c46d0dc90aaa-plugins-config-volume\") pod \"payload-processing-5bf47cd988-q7f2k\" (UID: \"4550ca18-d965-4ccb-a38e-c46d0dc90aaa\") " pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:56.054953 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.054787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxrh\" (UniqueName: \"kubernetes.io/projected/4550ca18-d965-4ccb-a38e-c46d0dc90aaa-kube-api-access-6vxrh\") pod \"payload-processing-5bf47cd988-q7f2k\" (UID: \"4550ca18-d965-4ccb-a38e-c46d0dc90aaa\") " pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:56.055472 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.055453 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4550ca18-d965-4ccb-a38e-c46d0dc90aaa-plugins-config-volume\") pod \"payload-processing-5bf47cd988-q7f2k\" (UID: \"4550ca18-d965-4ccb-a38e-c46d0dc90aaa\") " pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:56.062398 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.062370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxrh\" (UniqueName: \"kubernetes.io/projected/4550ca18-d965-4ccb-a38e-c46d0dc90aaa-kube-api-access-6vxrh\") pod \"payload-processing-5bf47cd988-q7f2k\" (UID: \"4550ca18-d965-4ccb-a38e-c46d0dc90aaa\") " pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:56.152403 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.152308 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:56.290508 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.290474 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-q7f2k"] Jul 19 07:52:56.294423 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:52:56.294392 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4550ca18_d965_4ccb_a38e_c46d0dc90aaa.slice/crio-e06a29414068ad8c5aafcf099476c941358267e41a136229f0f7a4972252a212 WatchSource:0}: Error finding container e06a29414068ad8c5aafcf099476c941358267e41a136229f0f7a4972252a212: Status 404 returned error can't find the container with id e06a29414068ad8c5aafcf099476c941358267e41a136229f0f7a4972252a212 Jul 19 07:52:56.423585 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.423493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" event={"ID":"95dcbaf9-aedd-4331-a0ef-0f247158d051","Type":"ContainerStarted","Data":"114d476f221143326bc04fb8be764e3ac68391bb4ca4eb500fca831230820f86"} Jul 19 07:52:56.424976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:56.424942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" event={"ID":"4550ca18-d965-4ccb-a38e-c46d0dc90aaa","Type":"ContainerStarted","Data":"e06a29414068ad8c5aafcf099476c941358267e41a136229f0f7a4972252a212"} Jul 19 07:52:57.434412 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:57.434273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" event={"ID":"95dcbaf9-aedd-4331-a0ef-0f247158d051","Type":"ContainerStarted","Data":"2f35477962d4a59752fde64b78d8258c86283b5757af6242b222e32762ce609b"} Jul 19 07:52:57.434939 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:57.434441 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:52:57.451223 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:57.451168 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" podStartSLOduration=1.046266153 podStartE2EDuration="2.451149804s" totalImagesPullingTime="1.404883651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:57.449965455 +0000 UTC m=+625.229628789" watchObservedRunningTime="2026-07-19 07:52:57.451149804 +0000 UTC m=+625.230813118" Jul 19 07:52:59.444461 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:59.444426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" event={"ID":"4550ca18-d965-4ccb-a38e-c46d0dc90aaa","Type":"ContainerStarted","Data":"b2faaee24de6b77e382abfe31a967bf43b866a2fb037f42c926fba64aedb3c5f"} Jul 19 07:52:59.444927 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:59.444506 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:52:59.460209 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:52:59.460158 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" podStartSLOduration=1.9346971 podStartE2EDuration="4.460141389s" totalImagesPullingTime="2.525444289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:59.458337968 +0000 UTC m=+627.238001303" watchObservedRunningTime="2026-07-19 07:52:59.460141389 +0000 UTC m=+627.239804703" Jul 19 07:53:03.445032 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:03.444998 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8694597d9b-n8b7c" Jul 19 07:53:10.452045 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:10.452013 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-q7f2k" Jul 19 07:53:25.298390 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.298352 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k"] Jul 19 07:53:25.301797 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.301773 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.303967 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.303946 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 19 07:53:25.304911 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.304889 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wc5bc\"" Jul 19 07:53:25.304981 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.304916 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 19 07:53:25.304981 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.304900 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 19 07:53:25.309480 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.309458 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k"] Jul 19 07:53:25.411377 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.411344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.411377 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.411377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.411638 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.411414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405e3ac0-4aa0-4242-a298-8959224e5eed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.411638 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.411525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.411638 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.411569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqlmj\" (UniqueName: \"kubernetes.io/projected/405e3ac0-4aa0-4242-a298-8959224e5eed-kube-api-access-sqlmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.411826 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.411671 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512355 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512553 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512553 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512553 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405e3ac0-4aa0-4242-a298-8959224e5eed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512553 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512553 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqlmj\" (UniqueName: \"kubernetes.io/projected/405e3ac0-4aa0-4242-a298-8959224e5eed-kube-api-access-sqlmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512925 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.512925 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.513024 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.512979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.514726 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.514710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405e3ac0-4aa0-4242-a298-8959224e5eed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.515125 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.515105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405e3ac0-4aa0-4242-a298-8959224e5eed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.520048 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.520022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqlmj\" (UniqueName: \"kubernetes.io/projected/405e3ac0-4aa0-4242-a298-8959224e5eed-kube-api-access-sqlmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k\" (UID: \"405e3ac0-4aa0-4242-a298-8959224e5eed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.612672 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.612634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:25.742529 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:25.742500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k"] Jul 19 07:53:25.745148 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:53:25.745122 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405e3ac0_4aa0_4242_a298_8959224e5eed.slice/crio-685dba421ad577ca0c0f7f8eed6e650187593449ac95e2edfa96978cfcec1233 WatchSource:0}: Error finding container 685dba421ad577ca0c0f7f8eed6e650187593449ac95e2edfa96978cfcec1233: Status 404 returned error can't find the container with id 685dba421ad577ca0c0f7f8eed6e650187593449ac95e2edfa96978cfcec1233 Jul 19 07:53:26.566256 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:26.566219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" event={"ID":"405e3ac0-4aa0-4242-a298-8959224e5eed","Type":"ContainerStarted","Data":"685dba421ad577ca0c0f7f8eed6e650187593449ac95e2edfa96978cfcec1233"} Jul 19 07:53:27.572172 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:27.572137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" event={"ID":"405e3ac0-4aa0-4242-a298-8959224e5eed","Type":"ContainerStarted","Data":"656ff1f14f327266f3ecaba309b8dfd2d86e8756df23682940919941c0ea67af"} Jul 19 07:53:27.590521 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:27.590462 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" podStartSLOduration=1.5318587830000001 podStartE2EDuration="2.59044695s" totalImagesPullingTime="1.058588167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:53:27.588458312 +0000 UTC m=+655.368121626" watchObservedRunningTime="2026-07-19 07:53:27.59044695 +0000 UTC m=+655.370110264" Jul 19 07:53:35.613051 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:35.613010 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:35.613507 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:35.613081 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:35.625768 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:35.625740 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:53:36.629868 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:53:36.629837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k" Jul 19 07:54:13.563793 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.563715 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b"] Jul 19 07:54:13.568870 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.568840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.571217 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.571192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 19 07:54:13.577495 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.577472 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b"] Jul 19 07:54:13.649877 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.649844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.650041 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.649895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.650041 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.649958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.650041 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.649977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.650041 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.650020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a508c8d7-6e00-4236-9aad-b334d0125402-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.650180 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.650042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdqz\" (UniqueName: \"kubernetes.io/projected/a508c8d7-6e00-4236-9aad-b334d0125402-kube-api-access-zfdqz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751268 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751268 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751495 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a508c8d7-6e00-4236-9aad-b334d0125402-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751495 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdqz\" (UniqueName: \"kubernetes.io/projected/a508c8d7-6e00-4236-9aad-b334d0125402-kube-api-access-zfdqz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751588 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751588 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751762 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751744 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751842 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.751924 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.751908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.753822 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.753801 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a508c8d7-6e00-4236-9aad-b334d0125402-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.753999 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.753981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a508c8d7-6e00-4236-9aad-b334d0125402-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.759047 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.759019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdqz\" (UniqueName: \"kubernetes.io/projected/a508c8d7-6e00-4236-9aad-b334d0125402-kube-api-access-zfdqz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wx64b\" (UID: \"a508c8d7-6e00-4236-9aad-b334d0125402\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:13.879354 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:13.879321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:14.017853 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:14.017820 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b"] Jul 19 07:54:14.023509 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:54:14.023478 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda508c8d7_6e00_4236_9aad_b334d0125402.slice/crio-b0453896c44691edd04a1e771bdb186006567c3a7bf42b2a7248783a138f3201 WatchSource:0}: Error finding container b0453896c44691edd04a1e771bdb186006567c3a7bf42b2a7248783a138f3201: Status 404 returned error can't find the container with id b0453896c44691edd04a1e771bdb186006567c3a7bf42b2a7248783a138f3201 Jul 19 07:54:14.024917 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:14.024897 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 07:54:14.788425 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:14.788394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" event={"ID":"a508c8d7-6e00-4236-9aad-b334d0125402","Type":"ContainerStarted","Data":"97fe1bfbdc2053e6a89396e61ee47ffcac3e36f68230d575b13aac4a7b7ff289"} Jul 19 07:54:14.788806 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:14.788450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" event={"ID":"a508c8d7-6e00-4236-9aad-b334d0125402","Type":"ContainerStarted","Data":"b0453896c44691edd04a1e771bdb186006567c3a7bf42b2a7248783a138f3201"} Jul 19 07:54:14.803136 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:14.803080 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" podStartSLOduration=1.636248765 podStartE2EDuration="1.803058977s" totalImagesPullingTime="166.810212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:54:14.80263619 +0000 UTC m=+702.582299507" watchObservedRunningTime="2026-07-19 07:54:14.803058977 +0000 UTC m=+702.582722295" Jul 19 07:54:23.879779 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:23.879684 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:23.879779 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:23.879752 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:23.892379 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:23.892350 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:24.842835 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:24.842806 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wx64b" Jul 19 07:54:38.474526 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.474493 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz"] Jul 19 07:54:38.478203 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.478159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.480543 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.480519 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 19 07:54:38.487002 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.486971 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz"] Jul 19 07:54:38.561410 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.561368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.561587 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.561418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.561587 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.561474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.561587 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.561518 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.561587 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.561547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68051fdf-91d0-4209-abbf-a7fb5ede2999-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.561776 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.561603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrn8\" (UniqueName: \"kubernetes.io/projected/68051fdf-91d0-4209-abbf-a7fb5ede2999-kube-api-access-4xrn8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.662914 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.662872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrn8\" (UniqueName: \"kubernetes.io/projected/68051fdf-91d0-4209-abbf-a7fb5ede2999-kube-api-access-4xrn8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.662945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.662976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.663000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.663020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.663038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68051fdf-91d0-4209-abbf-a7fb5ede2999-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663460 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.663435 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663546 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.663470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.663546 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.663512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.665442 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.665419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68051fdf-91d0-4209-abbf-a7fb5ede2999-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.665844 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.665824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68051fdf-91d0-4209-abbf-a7fb5ede2999-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.670634 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.670607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrn8\" (UniqueName: \"kubernetes.io/projected/68051fdf-91d0-4209-abbf-a7fb5ede2999-kube-api-access-4xrn8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz\" (UID: \"68051fdf-91d0-4209-abbf-a7fb5ede2999\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.788590 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.788502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:38.928443 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:38.928368 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz"] Jul 19 07:54:38.930906 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:54:38.930877 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68051fdf_91d0_4209_abbf_a7fb5ede2999.slice/crio-ba8243a6ef36898c7ad04e6168f92e135e752c35f9fdced3bd3998d3fe14a327 WatchSource:0}: Error finding container ba8243a6ef36898c7ad04e6168f92e135e752c35f9fdced3bd3998d3fe14a327: Status 404 returned error can't find the container with id ba8243a6ef36898c7ad04e6168f92e135e752c35f9fdced3bd3998d3fe14a327 Jul 19 07:54:39.901327 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:39.901287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" event={"ID":"68051fdf-91d0-4209-abbf-a7fb5ede2999","Type":"ContainerStarted","Data":"97008ca73c44b10f8cf0ed89395c171a4a934ab15bdc7ad640c67341df6f9eaf"} Jul 19 07:54:39.901787 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:39.901336 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" event={"ID":"68051fdf-91d0-4209-abbf-a7fb5ede2999","Type":"ContainerStarted","Data":"ba8243a6ef36898c7ad04e6168f92e135e752c35f9fdced3bd3998d3fe14a327"} Jul 19 07:54:39.917045 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:39.916998 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" podStartSLOduration=1.744912493 podStartE2EDuration="1.916982755s" totalImagesPullingTime="172.070262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:54:39.916421192 +0000 UTC m=+727.696084514" watchObservedRunningTime="2026-07-19 07:54:39.916982755 +0000 UTC m=+727.696646091" Jul 19 07:54:48.789465 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:48.789423 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:48.790137 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:48.789523 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:48.802015 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:48.801986 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:54:48.952002 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:54:48.951971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz" Jul 19 07:57:08.905330 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:08.905292 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-xktn7"] Jul 19 07:57:08.905875 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:08.905549 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58f4f5887c-xktn7" podUID="e112e106-3b10-485f-9adc-9740bae77219" containerName="manager" containerID="cri-o://237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683" gracePeriod=30 Jul 19 07:57:09.154849 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.154825 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:57:09.225275 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.225199 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e112e106-3b10-485f-9adc-9740bae77219-webhook-cert\") pod \"e112e106-3b10-485f-9adc-9740bae77219\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " Jul 19 07:57:09.225275 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.225246 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h7j2\" (UniqueName: \"kubernetes.io/projected/e112e106-3b10-485f-9adc-9740bae77219-kube-api-access-9h7j2\") pod \"e112e106-3b10-485f-9adc-9740bae77219\" (UID: \"e112e106-3b10-485f-9adc-9740bae77219\") " Jul 19 07:57:09.227388 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.227353 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e112e106-3b10-485f-9adc-9740bae77219-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e112e106-3b10-485f-9adc-9740bae77219" (UID: "e112e106-3b10-485f-9adc-9740bae77219"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:57:09.227388 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.227363 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e112e106-3b10-485f-9adc-9740bae77219-kube-api-access-9h7j2" (OuterVolumeSpecName: "kube-api-access-9h7j2") pod "e112e106-3b10-485f-9adc-9740bae77219" (UID: "e112e106-3b10-485f-9adc-9740bae77219"). InnerVolumeSpecName "kube-api-access-9h7j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:57:09.325880 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.325847 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7j2\" (UniqueName: \"kubernetes.io/projected/e112e106-3b10-485f-9adc-9740bae77219-kube-api-access-9h7j2\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:57:09.325880 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.325873 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e112e106-3b10-485f-9adc-9740bae77219-webhook-cert\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:57:09.581143 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.581051 2574 generic.go:356] "Generic (PLEG): container finished" podID="e112e106-3b10-485f-9adc-9740bae77219" containerID="237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683" exitCode=0 Jul 19 07:57:09.581143 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.581100 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-xktn7" Jul 19 07:57:09.581143 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.581117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-xktn7" event={"ID":"e112e106-3b10-485f-9adc-9740bae77219","Type":"ContainerDied","Data":"237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683"} Jul 19 07:57:09.581444 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.581168 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-xktn7" event={"ID":"e112e106-3b10-485f-9adc-9740bae77219","Type":"ContainerDied","Data":"ba7ce2c4d956dfa83493deab8d08b06de316a515d9ffe55a66de94b6a8b56f34"} Jul 19 07:57:09.581444 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.581192 2574 scope.go:117] "RemoveContainer" containerID="237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683" Jul 19 07:57:09.589887 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.589872 2574 scope.go:117] "RemoveContainer" containerID="237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683" Jul 19 07:57:09.590155 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:57:09.590137 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683\": container with ID starting with 237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683 not found: ID does not exist" containerID="237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683" Jul 19 07:57:09.590253 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.590161 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683"} err="failed to get container status \"237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683\": rpc error: code = NotFound desc = could not find container \"237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683\": container with ID starting with 237f00ec27d0959170cb7b3f22f1cf5d3387b8db17da3d3a4451e6b6b1ed7683 not found: ID does not exist" Jul 19 07:57:09.601443 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.601419 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-xktn7"] Jul 19 07:57:09.604706 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:09.604665 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-xktn7"] Jul 19 07:57:10.732329 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:57:10.732292 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e112e106-3b10-485f-9adc-9740bae77219" path="/var/lib/kubelet/pods/e112e106-3b10-485f-9adc-9740bae77219/volumes" Jul 19 07:58:10.377548 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.377515 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f4f5887c-fl7cm"] Jul 19 07:58:10.378018 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.377882 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e112e106-3b10-485f-9adc-9740bae77219" containerName="manager" Jul 19 07:58:10.378018 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.377894 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e112e106-3b10-485f-9adc-9740bae77219" containerName="manager" Jul 19 07:58:10.378018 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.377956 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e112e106-3b10-485f-9adc-9740bae77219" containerName="manager" Jul 19 07:58:10.386867 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.386840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.389869 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.389844 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 19 07:58:10.390777 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.390745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 19 07:58:10.390976 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.390950 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-fl7cm"] Jul 19 07:58:10.391091 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.390864 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w8hzw\"" Jul 19 07:58:10.468505 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.468467 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685092ad-9188-4a46-b8e5-8026a934cf93-webhook-cert\") pod \"maas-controller-58f4f5887c-fl7cm\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.468505 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.468508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvhw\" (UniqueName: \"kubernetes.io/projected/685092ad-9188-4a46-b8e5-8026a934cf93-kube-api-access-dzvhw\") pod \"maas-controller-58f4f5887c-fl7cm\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.570022 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.569977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685092ad-9188-4a46-b8e5-8026a934cf93-webhook-cert\") pod \"maas-controller-58f4f5887c-fl7cm\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.570022 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.570027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvhw\" (UniqueName: \"kubernetes.io/projected/685092ad-9188-4a46-b8e5-8026a934cf93-kube-api-access-dzvhw\") pod \"maas-controller-58f4f5887c-fl7cm\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.572576 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.572554 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685092ad-9188-4a46-b8e5-8026a934cf93-webhook-cert\") pod \"maas-controller-58f4f5887c-fl7cm\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.577297 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.577268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvhw\" (UniqueName: \"kubernetes.io/projected/685092ad-9188-4a46-b8e5-8026a934cf93-kube-api-access-dzvhw\") pod \"maas-controller-58f4f5887c-fl7cm\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.698916 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.698836 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:10.828829 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.828804 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-fl7cm"] Jul 19 07:58:10.831280 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:58:10.831252 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685092ad_9188_4a46_b8e5_8026a934cf93.slice/crio-16f6d3c330439ee8ede405f5cc84cc44879cfcec6ff82093dc0fdc27d77c845f WatchSource:0}: Error finding container 16f6d3c330439ee8ede405f5cc84cc44879cfcec6ff82093dc0fdc27d77c845f: Status 404 returned error can't find the container with id 16f6d3c330439ee8ede405f5cc84cc44879cfcec6ff82093dc0fdc27d77c845f Jul 19 07:58:10.861196 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:10.861161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" event={"ID":"685092ad-9188-4a46-b8e5-8026a934cf93","Type":"ContainerStarted","Data":"16f6d3c330439ee8ede405f5cc84cc44879cfcec6ff82093dc0fdc27d77c845f"} Jul 19 07:58:11.867246 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:11.867215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" event={"ID":"685092ad-9188-4a46-b8e5-8026a934cf93","Type":"ContainerStarted","Data":"2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77"} Jul 19 07:58:11.867629 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:11.867326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:11.883867 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:11.883821 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" podStartSLOduration=1.447633048 podStartE2EDuration="1.883805838s" totalImagesPullingTime="436.17279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:58:11.882485692 +0000 UTC m=+939.662149005" watchObservedRunningTime="2026-07-19 07:58:11.883805838 +0000 UTC m=+939.663469185" Jul 19 07:58:22.874473 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:22.874442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:27.503530 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.503487 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-fl7cm"] Jul 19 07:58:27.505856 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.503754 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" podUID="685092ad-9188-4a46-b8e5-8026a934cf93" containerName="manager" containerID="cri-o://2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77" gracePeriod=30 Jul 19 07:58:27.752002 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.751978 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:27.819294 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.819222 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzvhw\" (UniqueName: \"kubernetes.io/projected/685092ad-9188-4a46-b8e5-8026a934cf93-kube-api-access-dzvhw\") pod \"685092ad-9188-4a46-b8e5-8026a934cf93\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " Jul 19 07:58:27.819294 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.819256 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685092ad-9188-4a46-b8e5-8026a934cf93-webhook-cert\") pod \"685092ad-9188-4a46-b8e5-8026a934cf93\" (UID: \"685092ad-9188-4a46-b8e5-8026a934cf93\") " Jul 19 07:58:27.821512 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.821477 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685092ad-9188-4a46-b8e5-8026a934cf93-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "685092ad-9188-4a46-b8e5-8026a934cf93" (UID: "685092ad-9188-4a46-b8e5-8026a934cf93"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 07:58:27.821512 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.821489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685092ad-9188-4a46-b8e5-8026a934cf93-kube-api-access-dzvhw" (OuterVolumeSpecName: "kube-api-access-dzvhw") pod "685092ad-9188-4a46-b8e5-8026a934cf93" (UID: "685092ad-9188-4a46-b8e5-8026a934cf93"). InnerVolumeSpecName "kube-api-access-dzvhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 07:58:27.920115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.920081 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvhw\" (UniqueName: \"kubernetes.io/projected/685092ad-9188-4a46-b8e5-8026a934cf93-kube-api-access-dzvhw\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:58:27.920115 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.920108 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/685092ad-9188-4a46-b8e5-8026a934cf93-webhook-cert\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 07:58:27.948786 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.948759 2574 generic.go:356] "Generic (PLEG): container finished" podID="685092ad-9188-4a46-b8e5-8026a934cf93" containerID="2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77" exitCode=0 Jul 19 07:58:27.948933 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.948803 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" Jul 19 07:58:27.948933 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.948854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" event={"ID":"685092ad-9188-4a46-b8e5-8026a934cf93","Type":"ContainerDied","Data":"2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77"} Jul 19 07:58:27.948933 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.948903 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-fl7cm" event={"ID":"685092ad-9188-4a46-b8e5-8026a934cf93","Type":"ContainerDied","Data":"16f6d3c330439ee8ede405f5cc84cc44879cfcec6ff82093dc0fdc27d77c845f"} Jul 19 07:58:27.948933 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.948930 2574 scope.go:117] "RemoveContainer" containerID="2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77" Jul 19 07:58:27.963927 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.963907 2574 scope.go:117] "RemoveContainer" containerID="2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77" Jul 19 07:58:27.964220 ip-10-0-132-126 kubenswrapper[2574]: E0719 07:58:27.964203 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77\": container with ID starting with 2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77 not found: ID does not exist" containerID="2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77" Jul 19 07:58:27.964284 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.964231 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77"} err="failed to get container status \"2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77\": rpc error: code = NotFound desc = could not find container \"2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77\": container with ID starting with 2b7ec8f5bd6f1d6be26279e0a8b305ad31bade2d28f3c27b04d33bd14f3d1d77 not found: ID does not exist" Jul 19 07:58:27.973857 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.973833 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-fl7cm"] Jul 19 07:58:27.975265 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:27.975245 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-fl7cm"] Jul 19 07:58:28.716724 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.716687 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f4f5887c-8xcpd"] Jul 19 07:58:28.717087 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.717075 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685092ad-9188-4a46-b8e5-8026a934cf93" containerName="manager" Jul 19 07:58:28.717087 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.717086 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="685092ad-9188-4a46-b8e5-8026a934cf93" containerName="manager" Jul 19 07:58:28.717161 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.717143 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="685092ad-9188-4a46-b8e5-8026a934cf93" containerName="manager" Jul 19 07:58:28.721276 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.721253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:28.724233 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.724207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b1b6506-4b16-4b3a-98bb-45d6d616874e-webhook-cert\") pod \"maas-controller-58f4f5887c-8xcpd\" (UID: \"6b1b6506-4b16-4b3a-98bb-45d6d616874e\") " pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:28.724372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.724240 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w8hzw\"" Jul 19 07:58:28.724372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.724273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9wn\" (UniqueName: \"kubernetes.io/projected/6b1b6506-4b16-4b3a-98bb-45d6d616874e-kube-api-access-xf9wn\") pod \"maas-controller-58f4f5887c-8xcpd\" (UID: \"6b1b6506-4b16-4b3a-98bb-45d6d616874e\") " pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:28.724372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.724283 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 19 07:58:28.724372 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.724321 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 19 07:58:28.734576 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.734551 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685092ad-9188-4a46-b8e5-8026a934cf93" path="/var/lib/kubelet/pods/685092ad-9188-4a46-b8e5-8026a934cf93/volumes" Jul 19 07:58:28.734877 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.734859 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-8xcpd"] Jul 19 07:58:28.825125 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.825096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b1b6506-4b16-4b3a-98bb-45d6d616874e-webhook-cert\") pod \"maas-controller-58f4f5887c-8xcpd\" (UID: \"6b1b6506-4b16-4b3a-98bb-45d6d616874e\") " pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:28.825317 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.825159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9wn\" (UniqueName: \"kubernetes.io/projected/6b1b6506-4b16-4b3a-98bb-45d6d616874e-kube-api-access-xf9wn\") pod \"maas-controller-58f4f5887c-8xcpd\" (UID: \"6b1b6506-4b16-4b3a-98bb-45d6d616874e\") " pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:28.827734 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.827704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b1b6506-4b16-4b3a-98bb-45d6d616874e-webhook-cert\") pod \"maas-controller-58f4f5887c-8xcpd\" (UID: \"6b1b6506-4b16-4b3a-98bb-45d6d616874e\") " pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:28.832636 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:28.832608 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9wn\" (UniqueName: \"kubernetes.io/projected/6b1b6506-4b16-4b3a-98bb-45d6d616874e-kube-api-access-xf9wn\") pod \"maas-controller-58f4f5887c-8xcpd\" (UID: \"6b1b6506-4b16-4b3a-98bb-45d6d616874e\") " pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:29.032642 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:29.032561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:29.162269 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:29.162192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f4f5887c-8xcpd"] Jul 19 07:58:29.165149 ip-10-0-132-126 kubenswrapper[2574]: W0719 07:58:29.165121 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1b6506_4b16_4b3a_98bb_45d6d616874e.slice/crio-2591e32825d3d61d46524737c0998ea18e0f98b4da7bd7ba92a54db69fe431be WatchSource:0}: Error finding container 2591e32825d3d61d46524737c0998ea18e0f98b4da7bd7ba92a54db69fe431be: Status 404 returned error can't find the container with id 2591e32825d3d61d46524737c0998ea18e0f98b4da7bd7ba92a54db69fe431be Jul 19 07:58:29.960709 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:29.960597 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" event={"ID":"6b1b6506-4b16-4b3a-98bb-45d6d616874e","Type":"ContainerStarted","Data":"8ad1d9edfa9069f1c2c1eede648c2628ca8b3271730026c07bd015a81a9418df"} Jul 19 07:58:29.960709 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:29.960652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" event={"ID":"6b1b6506-4b16-4b3a-98bb-45d6d616874e","Type":"ContainerStarted","Data":"2591e32825d3d61d46524737c0998ea18e0f98b4da7bd7ba92a54db69fe431be"} Jul 19 07:58:29.960709 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:29.960706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 07:58:29.977734 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:29.977688 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" podStartSLOduration=1.508413511 podStartE2EDuration="1.977657749s" totalImagesPullingTime="469.244238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:58:29.975871699 +0000 UTC m=+957.755535014" watchObservedRunningTime="2026-07-19 07:58:29.977657749 +0000 UTC m=+957.757321063" Jul 19 07:58:40.968004 ip-10-0-132-126 kubenswrapper[2574]: I0719 07:58:40.967974 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f4f5887c-8xcpd" Jul 19 08:15:00.131574 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.131535 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n"] Jul 19 08:15:00.135059 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.135042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" Jul 19 08:15:00.137311 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.137290 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:15:00.145095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.145057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n"] Jul 19 08:15:00.147139 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.147120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" Jul 19 08:15:00.276637 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.276611 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n"] Jul 19 08:15:00.279110 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:15:00.279083 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9c1c6c_b743_4afa_a8de_736e04c9f517.slice/crio-a60569b871b2386a9ffb1abcc76f5f87f685e32d4c24384740bdbaba1524d24c WatchSource:0}: Error finding container a60569b871b2386a9ffb1abcc76f5f87f685e32d4c24384740bdbaba1524d24c: Status 404 returned error can't find the container with id a60569b871b2386a9ffb1abcc76f5f87f685e32d4c24384740bdbaba1524d24c Jul 19 08:15:00.280947 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.280931 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:15:00.417770 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:00.417657 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" event={"ID":"4d9c1c6c-b743-4afa-a8de-736e04c9f517","Type":"ContainerStarted","Data":"a60569b871b2386a9ffb1abcc76f5f87f685e32d4c24384740bdbaba1524d24c"} Jul 19 08:15:03.435094 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:03.435055 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerID="746acabca4f6d561c6f45053158f96f964127e448c7f1b23ebb09898e6dc2bc7" exitCode=7 Jul 19 08:15:03.435501 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:03.435114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" event={"ID":"4d9c1c6c-b743-4afa-a8de-736e04c9f517","Type":"ContainerDied","Data":"746acabca4f6d561c6f45053158f96f964127e448c7f1b23ebb09898e6dc2bc7"} Jul 19 08:15:03.435501 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:03.435311 2574 scope.go:117] "RemoveContainer" containerID="746acabca4f6d561c6f45053158f96f964127e448c7f1b23ebb09898e6dc2bc7" Jul 19 08:15:04.441782 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:04.441745 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerID="b0b5982c8d15d583185f7c0221fcf657908b2ec6c6ce0ea5822964a8ef522008" exitCode=7 Jul 19 08:15:04.442205 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:04.441804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" event={"ID":"4d9c1c6c-b743-4afa-a8de-736e04c9f517","Type":"ContainerDied","Data":"b0b5982c8d15d583185f7c0221fcf657908b2ec6c6ce0ea5822964a8ef522008"} Jul 19 08:15:04.442205 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:04.441879 2574 scope.go:117] "RemoveContainer" containerID="746acabca4f6d561c6f45053158f96f964127e448c7f1b23ebb09898e6dc2bc7" Jul 19 08:15:04.442205 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:04.442034 2574 scope.go:117] "RemoveContainer" containerID="b0b5982c8d15d583185f7c0221fcf657908b2ec6c6ce0ea5822964a8ef522008" Jul 19 08:15:04.442335 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:15:04.442280 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n_odh-ai-gateway-infra(4d9c1c6c-b743-4afa-a8de-736e04c9f517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" Jul 19 08:15:05.447833 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:05.447809 2574 scope.go:117] "RemoveContainer" containerID="b0b5982c8d15d583185f7c0221fcf657908b2ec6c6ce0ea5822964a8ef522008" Jul 19 08:15:05.448232 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:15:05.448039 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n_odh-ai-gateway-infra(4d9c1c6c-b743-4afa-a8de-736e04c9f517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" Jul 19 08:15:18.726428 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:18.726394 2574 scope.go:117] "RemoveContainer" containerID="b0b5982c8d15d583185f7c0221fcf657908b2ec6c6ce0ea5822964a8ef522008" Jul 19 08:15:19.517247 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:19.517215 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerID="7daedf22bb53ee84477bdc866f1a9efb8d2a8bda8e6cb8246d65638a63fba71d" exitCode=7 Jul 19 08:15:19.517419 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:19.517272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" event={"ID":"4d9c1c6c-b743-4afa-a8de-736e04c9f517","Type":"ContainerDied","Data":"7daedf22bb53ee84477bdc866f1a9efb8d2a8bda8e6cb8246d65638a63fba71d"} Jul 19 08:15:19.517419 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:19.517324 2574 scope.go:117] "RemoveContainer" containerID="b0b5982c8d15d583185f7c0221fcf657908b2ec6c6ce0ea5822964a8ef522008" Jul 19 08:15:19.517618 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:19.517596 2574 scope.go:117] "RemoveContainer" containerID="7daedf22bb53ee84477bdc866f1a9efb8d2a8bda8e6cb8246d65638a63fba71d" Jul 19 08:15:19.517889 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:15:19.517866 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n_odh-ai-gateway-infra(4d9c1c6c-b743-4afa-a8de-736e04c9f517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" Jul 19 08:15:20.542164 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:20.542048 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n"] Jul 19 08:15:20.677461 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:20.677429 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" Jul 19 08:15:21.529611 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:21.529575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" event={"ID":"4d9c1c6c-b743-4afa-a8de-736e04c9f517","Type":"ContainerDied","Data":"a60569b871b2386a9ffb1abcc76f5f87f685e32d4c24384740bdbaba1524d24c"} Jul 19 08:15:21.529845 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:21.529634 2574 scope.go:117] "RemoveContainer" containerID="7daedf22bb53ee84477bdc866f1a9efb8d2a8bda8e6cb8246d65638a63fba71d" Jul 19 08:15:21.529845 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:21.529593 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n" Jul 19 08:15:21.548641 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:21.548614 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n"] Jul 19 08:15:21.554003 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:21.553980 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2a60bf57-29740815-v6c2n"] Jul 19 08:15:22.732890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:22.732848 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" path="/var/lib/kubelet/pods/4d9c1c6c-b743-4afa-a8de-736e04c9f517/volumes" Jul 19 08:15:52.686104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686070 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp"] Jul 19 08:15:52.686643 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686624 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:15:52.686706 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686647 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:15:52.686706 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686667 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:15:52.686706 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686693 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:15:52.686815 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686790 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:15:52.686815 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.686801 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:15:52.690231 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.690210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:15:52.692614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.692590 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:15:52.696167 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.696142 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp"] Jul 19 08:15:52.828070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.828021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d41951-e935-4916-a8e7-b70b3aa480f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp\" (UID: \"c7d41951-e935-4916-a8e7-b70b3aa480f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:15:52.929616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.929558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d41951-e935-4916-a8e7-b70b3aa480f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp\" (UID: \"c7d41951-e935-4916-a8e7-b70b3aa480f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:15:52.930234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:52.930213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d41951-e935-4916-a8e7-b70b3aa480f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp\" (UID: \"c7d41951-e935-4916-a8e7-b70b3aa480f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:15:53.003716 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:53.003611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:15:53.138219 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:53.138150 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp"] Jul 19 08:15:53.141062 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:15:53.141032 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d41951_e935_4916_a8e7_b70b3aa480f7.slice/crio-a14c0b869112e1ed11f9b8dba13cd5d69950a10453a86cd7709befda99b3944c WatchSource:0}: Error finding container a14c0b869112e1ed11f9b8dba13cd5d69950a10453a86cd7709befda99b3944c: Status 404 returned error can't find the container with id a14c0b869112e1ed11f9b8dba13cd5d69950a10453a86cd7709befda99b3944c Jul 19 08:15:53.684423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:53.684389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" event={"ID":"c7d41951-e935-4916-a8e7-b70b3aa480f7","Type":"ContainerStarted","Data":"0eb5963a12ae914d09a3d1d6a12944410687ef0adc7a711a535b8a73a4e37d29"} Jul 19 08:15:53.684605 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:53.684441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" event={"ID":"c7d41951-e935-4916-a8e7-b70b3aa480f7","Type":"ContainerStarted","Data":"a14c0b869112e1ed11f9b8dba13cd5d69950a10453a86cd7709befda99b3944c"} Jul 19 08:15:53.700247 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:53.700203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" podStartSLOduration=1.70018649 podStartE2EDuration="1.70018649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:15:53.698771494 +0000 UTC m=+2001.478434834" watchObservedRunningTime="2026-07-19 08:15:53.70018649 +0000 UTC m=+2001.479849804" Jul 19 08:15:54.691214 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:54.691174 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerID="0eb5963a12ae914d09a3d1d6a12944410687ef0adc7a711a535b8a73a4e37d29" exitCode=7 Jul 19 08:15:54.691385 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:54.691225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" event={"ID":"c7d41951-e935-4916-a8e7-b70b3aa480f7","Type":"ContainerDied","Data":"0eb5963a12ae914d09a3d1d6a12944410687ef0adc7a711a535b8a73a4e37d29"} Jul 19 08:15:54.691427 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:54.691418 2574 scope.go:117] "RemoveContainer" containerID="0eb5963a12ae914d09a3d1d6a12944410687ef0adc7a711a535b8a73a4e37d29" Jul 19 08:15:55.698093 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:55.698052 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerID="2a53445b6d11408e6601de9152cb9af4437635bc3a20a3cc9bf5dff6b0ee080d" exitCode=7 Jul 19 08:15:55.698093 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:55.698091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" event={"ID":"c7d41951-e935-4916-a8e7-b70b3aa480f7","Type":"ContainerDied","Data":"2a53445b6d11408e6601de9152cb9af4437635bc3a20a3cc9bf5dff6b0ee080d"} Jul 19 08:15:55.698603 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:55.698138 2574 scope.go:117] "RemoveContainer" containerID="0eb5963a12ae914d09a3d1d6a12944410687ef0adc7a711a535b8a73a4e37d29" Jul 19 08:15:55.698603 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:55.698369 2574 scope.go:117] "RemoveContainer" containerID="2a53445b6d11408e6601de9152cb9af4437635bc3a20a3cc9bf5dff6b0ee080d" Jul 19 08:15:55.698702 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:15:55.698597 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp_odh-ai-gateway-infra(c7d41951-e935-4916-a8e7-b70b3aa480f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" Jul 19 08:15:56.704565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:15:56.704538 2574 scope.go:117] "RemoveContainer" containerID="2a53445b6d11408e6601de9152cb9af4437635bc3a20a3cc9bf5dff6b0ee080d" Jul 19 08:15:56.704993 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:15:56.704744 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp_odh-ai-gateway-infra(c7d41951-e935-4916-a8e7-b70b3aa480f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" Jul 19 08:16:10.726863 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:10.726830 2574 scope.go:117] "RemoveContainer" containerID="2a53445b6d11408e6601de9152cb9af4437635bc3a20a3cc9bf5dff6b0ee080d" Jul 19 08:16:11.772246 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:11.772209 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerID="b1bac4312a7973f65e1527239788f3619f4e141d9b6e4ce589c26fea08bb2264" exitCode=7 Jul 19 08:16:11.772656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:11.772278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" event={"ID":"c7d41951-e935-4916-a8e7-b70b3aa480f7","Type":"ContainerDied","Data":"b1bac4312a7973f65e1527239788f3619f4e141d9b6e4ce589c26fea08bb2264"} Jul 19 08:16:11.772656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:11.772366 2574 scope.go:117] "RemoveContainer" containerID="2a53445b6d11408e6601de9152cb9af4437635bc3a20a3cc9bf5dff6b0ee080d" Jul 19 08:16:11.772792 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:11.772765 2574 scope.go:117] "RemoveContainer" containerID="b1bac4312a7973f65e1527239788f3619f4e141d9b6e4ce589c26fea08bb2264" Jul 19 08:16:11.773006 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:11.772987 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp_odh-ai-gateway-infra(c7d41951-e935-4916-a8e7-b70b3aa480f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" Jul 19 08:16:12.794845 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:12.794809 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp"] Jul 19 08:16:12.932057 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:12.932033 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:16:13.010293 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.010257 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d41951-e935-4916-a8e7-b70b3aa480f7-maas-api-service-ca\") pod \"c7d41951-e935-4916-a8e7-b70b3aa480f7\" (UID: \"c7d41951-e935-4916-a8e7-b70b3aa480f7\") " Jul 19 08:16:13.010652 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.010626 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d41951-e935-4916-a8e7-b70b3aa480f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7d41951-e935-4916-a8e7-b70b3aa480f7" (UID: "c7d41951-e935-4916-a8e7-b70b3aa480f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:16:13.111109 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.111074 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d41951-e935-4916-a8e7-b70b3aa480f7-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:16:13.784365 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.784327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" event={"ID":"c7d41951-e935-4916-a8e7-b70b3aa480f7","Type":"ContainerDied","Data":"a14c0b869112e1ed11f9b8dba13cd5d69950a10453a86cd7709befda99b3944c"} Jul 19 08:16:13.784365 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.784367 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp" Jul 19 08:16:13.784593 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.784391 2574 scope.go:117] "RemoveContainer" containerID="b1bac4312a7973f65e1527239788f3619f4e141d9b6e4ce589c26fea08bb2264" Jul 19 08:16:13.805239 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.805204 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp"] Jul 19 08:16:13.811402 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:13.811376 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5g2xp"] Jul 19 08:16:14.518164 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518130 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46"] Jul 19 08:16:14.518527 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518513 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:16:14.518577 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518529 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:16:14.518577 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518547 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:14.518577 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518553 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:14.518577 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518569 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:14.518577 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518574 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:14.518760 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518622 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:14.518760 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518631 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:14.518760 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.518640 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9c1c6c-b743-4afa-a8de-736e04c9f517" containerName="cleanup" Jul 19 08:16:14.522837 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.522820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:14.525335 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.525308 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:16:14.527376 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.527352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46"] Jul 19 08:16:14.621994 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.621960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35762fa-7228-4c39-9bcd-243db03da831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46\" (UID: \"a35762fa-7228-4c39-9bcd-243db03da831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:14.723321 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.723281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35762fa-7228-4c39-9bcd-243db03da831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46\" (UID: \"a35762fa-7228-4c39-9bcd-243db03da831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:14.723916 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.723898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35762fa-7228-4c39-9bcd-243db03da831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46\" (UID: \"a35762fa-7228-4c39-9bcd-243db03da831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:14.732032 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.732007 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" path="/var/lib/kubelet/pods/c7d41951-e935-4916-a8e7-b70b3aa480f7/volumes" Jul 19 08:16:14.835446 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.835418 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:14.974400 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:14.974376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46"] Jul 19 08:16:14.976649 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:16:14.976615 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35762fa_7228_4c39_9bcd_243db03da831.slice/crio-e54d67c715fe78d930897625ec907378a5af9ca6d5158960cfaf61078939ba21 WatchSource:0}: Error finding container e54d67c715fe78d930897625ec907378a5af9ca6d5158960cfaf61078939ba21: Status 404 returned error can't find the container with id e54d67c715fe78d930897625ec907378a5af9ca6d5158960cfaf61078939ba21 Jul 19 08:16:15.796081 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:15.796044 2574 generic.go:356] "Generic (PLEG): container finished" podID="a35762fa-7228-4c39-9bcd-243db03da831" containerID="955a22c0a3fb44225ccf63f98ad8a034b4a995fdaa5fd6afb410a92fd6d0c393" exitCode=6 Jul 19 08:16:15.796270 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:15.796139 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" event={"ID":"a35762fa-7228-4c39-9bcd-243db03da831","Type":"ContainerDied","Data":"955a22c0a3fb44225ccf63f98ad8a034b4a995fdaa5fd6afb410a92fd6d0c393"} Jul 19 08:16:15.796270 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:15.796196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" event={"ID":"a35762fa-7228-4c39-9bcd-243db03da831","Type":"ContainerStarted","Data":"e54d67c715fe78d930897625ec907378a5af9ca6d5158960cfaf61078939ba21"} Jul 19 08:16:15.796377 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:15.796360 2574 scope.go:117] "RemoveContainer" containerID="955a22c0a3fb44225ccf63f98ad8a034b4a995fdaa5fd6afb410a92fd6d0c393" Jul 19 08:16:16.802534 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:16.802447 2574 generic.go:356] "Generic (PLEG): container finished" podID="a35762fa-7228-4c39-9bcd-243db03da831" containerID="92e51a13673f237d5816e956c727522d2ac88d9ad61b793bb540dc034f872d35" exitCode=6 Jul 19 08:16:16.802534 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:16.802506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" event={"ID":"a35762fa-7228-4c39-9bcd-243db03da831","Type":"ContainerDied","Data":"92e51a13673f237d5816e956c727522d2ac88d9ad61b793bb540dc034f872d35"} Jul 19 08:16:16.802997 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:16.802561 2574 scope.go:117] "RemoveContainer" containerID="955a22c0a3fb44225ccf63f98ad8a034b4a995fdaa5fd6afb410a92fd6d0c393" Jul 19 08:16:16.802997 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:16.802783 2574 scope.go:117] "RemoveContainer" containerID="92e51a13673f237d5816e956c727522d2ac88d9ad61b793bb540dc034f872d35" Jul 19 08:16:16.803091 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:16.803031 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46_odh-ai-gateway-infra(a35762fa-7228-4c39-9bcd-243db03da831)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" podUID="a35762fa-7228-4c39-9bcd-243db03da831" Jul 19 08:16:17.809883 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:17.809852 2574 scope.go:117] "RemoveContainer" containerID="92e51a13673f237d5816e956c727522d2ac88d9ad61b793bb540dc034f872d35" Jul 19 08:16:17.810268 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:17.810076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46_odh-ai-gateway-infra(a35762fa-7228-4c39-9bcd-243db03da831)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" podUID="a35762fa-7228-4c39-9bcd-243db03da831" Jul 19 08:16:31.726256 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:31.726222 2574 scope.go:117] "RemoveContainer" containerID="92e51a13673f237d5816e956c727522d2ac88d9ad61b793bb540dc034f872d35" Jul 19 08:16:32.883476 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:32.883439 2574 generic.go:356] "Generic (PLEG): container finished" podID="a35762fa-7228-4c39-9bcd-243db03da831" containerID="57c2c7bd8ba2e27cb95eb583d2ea8575ab4aab960463eb422711dcb62f62bf2f" exitCode=6 Jul 19 08:16:32.883875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:32.883507 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" event={"ID":"a35762fa-7228-4c39-9bcd-243db03da831","Type":"ContainerDied","Data":"57c2c7bd8ba2e27cb95eb583d2ea8575ab4aab960463eb422711dcb62f62bf2f"} Jul 19 08:16:32.883875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:32.883567 2574 scope.go:117] "RemoveContainer" containerID="92e51a13673f237d5816e956c727522d2ac88d9ad61b793bb540dc034f872d35" Jul 19 08:16:32.884014 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:32.883911 2574 scope.go:117] "RemoveContainer" containerID="57c2c7bd8ba2e27cb95eb583d2ea8575ab4aab960463eb422711dcb62f62bf2f" Jul 19 08:16:32.884167 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:32.884146 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46_odh-ai-gateway-infra(a35762fa-7228-4c39-9bcd-243db03da831)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" podUID="a35762fa-7228-4c39-9bcd-243db03da831" Jul 19 08:16:33.906854 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:33.906817 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46"] Jul 19 08:16:34.043387 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.043365 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:34.102284 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.102251 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35762fa-7228-4c39-9bcd-243db03da831-maas-api-service-ca\") pod \"a35762fa-7228-4c39-9bcd-243db03da831\" (UID: \"a35762fa-7228-4c39-9bcd-243db03da831\") " Jul 19 08:16:34.102616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.102589 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35762fa-7228-4c39-9bcd-243db03da831-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a35762fa-7228-4c39-9bcd-243db03da831" (UID: "a35762fa-7228-4c39-9bcd-243db03da831"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:16:34.203141 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.203050 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35762fa-7228-4c39-9bcd-243db03da831-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:16:34.895736 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.895707 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" Jul 19 08:16:34.895898 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.895732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46" event={"ID":"a35762fa-7228-4c39-9bcd-243db03da831","Type":"ContainerDied","Data":"e54d67c715fe78d930897625ec907378a5af9ca6d5158960cfaf61078939ba21"} Jul 19 08:16:34.895898 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.895824 2574 scope.go:117] "RemoveContainer" containerID="57c2c7bd8ba2e27cb95eb583d2ea8575ab4aab960463eb422711dcb62f62bf2f" Jul 19 08:16:34.910887 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.910858 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46"] Jul 19 08:16:34.916295 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:34.916264 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lrd46"] Jul 19 08:16:36.731648 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:36.731610 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a35762fa-7228-4c39-9bcd-243db03da831" path="/var/lib/kubelet/pods/a35762fa-7228-4c39-9bcd-243db03da831/volumes" Jul 19 08:16:46.837988 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.837955 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt"] Jul 19 08:16:46.838390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838346 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838358 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838373 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:46.838390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838378 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:46.838390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838393 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838399 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838404 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838452 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838460 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:46.838753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.838467 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d41951-e935-4916-a8e7-b70b3aa480f7" containerName="revoke-keys" Jul 19 08:16:46.842391 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.842374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:16:46.845369 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.845346 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:16:46.848360 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.848339 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt"] Jul 19 08:16:46.923597 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:46.923561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/618bb089-02a4-4258-b8ca-69e34763bf1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt\" (UID: \"618bb089-02a4-4258-b8ca-69e34763bf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:16:47.024756 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.024715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/618bb089-02a4-4258-b8ca-69e34763bf1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt\" (UID: \"618bb089-02a4-4258-b8ca-69e34763bf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:16:47.025349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.025327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/618bb089-02a4-4258-b8ca-69e34763bf1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt\" (UID: \"618bb089-02a4-4258-b8ca-69e34763bf1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:16:47.153738 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.153630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:16:47.293957 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.293922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt"] Jul 19 08:16:47.297841 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:16:47.297815 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618bb089_02a4_4258_b8ca_69e34763bf1a.slice/crio-4fa54a25986f1a054f26e9e855374983ea034d2d2ef2cf5a7776c197c51c5ed7 WatchSource:0}: Error finding container 4fa54a25986f1a054f26e9e855374983ea034d2d2ef2cf5a7776c197c51c5ed7: Status 404 returned error can't find the container with id 4fa54a25986f1a054f26e9e855374983ea034d2d2ef2cf5a7776c197c51c5ed7 Jul 19 08:16:47.956825 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.956788 2574 generic.go:356] "Generic (PLEG): container finished" podID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerID="453f60d53d16fbeda2af695eb99f36c187067793a5c4b62579fe3a0661817cbf" exitCode=6 Jul 19 08:16:47.957294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.956882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" event={"ID":"618bb089-02a4-4258-b8ca-69e34763bf1a","Type":"ContainerDied","Data":"453f60d53d16fbeda2af695eb99f36c187067793a5c4b62579fe3a0661817cbf"} Jul 19 08:16:47.957294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.956934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" event={"ID":"618bb089-02a4-4258-b8ca-69e34763bf1a","Type":"ContainerStarted","Data":"4fa54a25986f1a054f26e9e855374983ea034d2d2ef2cf5a7776c197c51c5ed7"} Jul 19 08:16:47.957294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:47.957110 2574 scope.go:117] "RemoveContainer" containerID="453f60d53d16fbeda2af695eb99f36c187067793a5c4b62579fe3a0661817cbf" Jul 19 08:16:48.963346 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:48.963307 2574 generic.go:356] "Generic (PLEG): container finished" podID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerID="cef8349d2e34f4492d94b4f5082cec96c50156a1f933da492a7a95b8caca53eb" exitCode=6 Jul 19 08:16:48.963903 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:48.963393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" event={"ID":"618bb089-02a4-4258-b8ca-69e34763bf1a","Type":"ContainerDied","Data":"cef8349d2e34f4492d94b4f5082cec96c50156a1f933da492a7a95b8caca53eb"} Jul 19 08:16:48.963903 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:48.963451 2574 scope.go:117] "RemoveContainer" containerID="453f60d53d16fbeda2af695eb99f36c187067793a5c4b62579fe3a0661817cbf" Jul 19 08:16:48.963903 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:48.963658 2574 scope.go:117] "RemoveContainer" containerID="cef8349d2e34f4492d94b4f5082cec96c50156a1f933da492a7a95b8caca53eb" Jul 19 08:16:48.964084 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:48.963911 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt_odh-ai-gateway-infra(618bb089-02a4-4258-b8ca-69e34763bf1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" Jul 19 08:16:49.969473 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:49.969438 2574 scope.go:117] "RemoveContainer" containerID="cef8349d2e34f4492d94b4f5082cec96c50156a1f933da492a7a95b8caca53eb" Jul 19 08:16:49.969875 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:49.969657 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt_odh-ai-gateway-infra(618bb089-02a4-4258-b8ca-69e34763bf1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" Jul 19 08:16:52.733409 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.733370 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd"] Jul 19 08:16:52.733841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.733825 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35762fa-7228-4c39-9bcd-243db03da831" containerName="revoke-keys" Jul 19 08:16:52.738143 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.738127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:16:52.745847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.745426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd"] Jul 19 08:16:52.880739 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.880645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac174559-8310-41c4-b2e4-e73e09a392a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd\" (UID: \"ac174559-8310-41c4-b2e4-e73e09a392a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:16:52.982117 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.982076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac174559-8310-41c4-b2e4-e73e09a392a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd\" (UID: \"ac174559-8310-41c4-b2e4-e73e09a392a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:16:52.982748 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:52.982723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac174559-8310-41c4-b2e4-e73e09a392a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd\" (UID: \"ac174559-8310-41c4-b2e4-e73e09a392a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:16:53.049847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:53.049749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:16:53.187197 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:53.187163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd"] Jul 19 08:16:53.190383 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:16:53.190355 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac174559_8310_41c4_b2e4_e73e09a392a9.slice/crio-7015ecc839ea49d263ba0b1d4fa0f5ab3f8ed4324bff50568fd936cb889b76cf WatchSource:0}: Error finding container 7015ecc839ea49d263ba0b1d4fa0f5ab3f8ed4324bff50568fd936cb889b76cf: Status 404 returned error can't find the container with id 7015ecc839ea49d263ba0b1d4fa0f5ab3f8ed4324bff50568fd936cb889b76cf Jul 19 08:16:53.987715 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:53.987653 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerID="0b6cf1234120169d901a5c7bf71ebe9e74de802fb3720f6a3dd7fdd9429bbeed" exitCode=7 Jul 19 08:16:53.988099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:53.987739 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" event={"ID":"ac174559-8310-41c4-b2e4-e73e09a392a9","Type":"ContainerDied","Data":"0b6cf1234120169d901a5c7bf71ebe9e74de802fb3720f6a3dd7fdd9429bbeed"} Jul 19 08:16:53.988099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:53.987798 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" event={"ID":"ac174559-8310-41c4-b2e4-e73e09a392a9","Type":"ContainerStarted","Data":"7015ecc839ea49d263ba0b1d4fa0f5ab3f8ed4324bff50568fd936cb889b76cf"} Jul 19 08:16:53.988099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:53.987993 2574 scope.go:117] "RemoveContainer" containerID="0b6cf1234120169d901a5c7bf71ebe9e74de802fb3720f6a3dd7fdd9429bbeed" Jul 19 08:16:54.994443 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:54.994406 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerID="6f2d87544a0d9cfd85e5c40eabf16849d6447f214e7e2d14183035ccf776d199" exitCode=7 Jul 19 08:16:54.994943 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:54.994491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" event={"ID":"ac174559-8310-41c4-b2e4-e73e09a392a9","Type":"ContainerDied","Data":"6f2d87544a0d9cfd85e5c40eabf16849d6447f214e7e2d14183035ccf776d199"} Jul 19 08:16:54.994943 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:54.994558 2574 scope.go:117] "RemoveContainer" containerID="0b6cf1234120169d901a5c7bf71ebe9e74de802fb3720f6a3dd7fdd9429bbeed" Jul 19 08:16:54.994943 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:54.994817 2574 scope.go:117] "RemoveContainer" containerID="6f2d87544a0d9cfd85e5c40eabf16849d6447f214e7e2d14183035ccf776d199" Jul 19 08:16:54.995126 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:54.995105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd_odh-ai-gateway-infra(ac174559-8310-41c4-b2e4-e73e09a392a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" Jul 19 08:16:56.002117 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:16:56.002084 2574 scope.go:117] "RemoveContainer" containerID="6f2d87544a0d9cfd85e5c40eabf16849d6447f214e7e2d14183035ccf776d199" Jul 19 08:16:56.002501 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:16:56.002311 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd_odh-ai-gateway-infra(ac174559-8310-41c4-b2e4-e73e09a392a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" Jul 19 08:17:00.727068 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:00.727030 2574 scope.go:117] "RemoveContainer" containerID="cef8349d2e34f4492d94b4f5082cec96c50156a1f933da492a7a95b8caca53eb" Jul 19 08:17:01.026634 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:01.026528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" event={"ID":"618bb089-02a4-4258-b8ca-69e34763bf1a","Type":"ContainerStarted","Data":"f51405b52f82c73d104674390284cefdb1278b1310fbf6560f051b7998f8e62f"} Jul 19 08:17:01.042918 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:01.042862 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" podStartSLOduration=15.0428447 podStartE2EDuration="15.0428447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:17:01.041398282 +0000 UTC m=+2068.821061598" watchObservedRunningTime="2026-07-19 08:17:01.0428447 +0000 UTC m=+2068.822508014" Jul 19 08:17:02.033505 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:02.033470 2574 generic.go:356] "Generic (PLEG): container finished" podID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerID="f51405b52f82c73d104674390284cefdb1278b1310fbf6560f051b7998f8e62f" exitCode=6 Jul 19 08:17:02.034034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:02.033540 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" event={"ID":"618bb089-02a4-4258-b8ca-69e34763bf1a","Type":"ContainerDied","Data":"f51405b52f82c73d104674390284cefdb1278b1310fbf6560f051b7998f8e62f"} Jul 19 08:17:02.034034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:02.033600 2574 scope.go:117] "RemoveContainer" containerID="cef8349d2e34f4492d94b4f5082cec96c50156a1f933da492a7a95b8caca53eb" Jul 19 08:17:02.034034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:02.033881 2574 scope.go:117] "RemoveContainer" containerID="f51405b52f82c73d104674390284cefdb1278b1310fbf6560f051b7998f8e62f" Jul 19 08:17:02.034201 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:02.034089 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt_odh-ai-gateway-infra(618bb089-02a4-4258-b8ca-69e34763bf1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" Jul 19 08:17:02.057559 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:02.057525 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt"] Jul 19 08:17:03.165316 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:03.165289 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:17:03.276956 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:03.276925 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/618bb089-02a4-4258-b8ca-69e34763bf1a-maas-api-service-ca\") pod \"618bb089-02a4-4258-b8ca-69e34763bf1a\" (UID: \"618bb089-02a4-4258-b8ca-69e34763bf1a\") " Jul 19 08:17:03.277254 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:03.277233 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618bb089-02a4-4258-b8ca-69e34763bf1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "618bb089-02a4-4258-b8ca-69e34763bf1a" (UID: "618bb089-02a4-4258-b8ca-69e34763bf1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:17:03.378586 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:03.378550 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/618bb089-02a4-4258-b8ca-69e34763bf1a-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:17:04.045394 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:04.045358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" event={"ID":"618bb089-02a4-4258-b8ca-69e34763bf1a","Type":"ContainerDied","Data":"4fa54a25986f1a054f26e9e855374983ea034d2d2ef2cf5a7776c197c51c5ed7"} Jul 19 08:17:04.045601 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:04.045420 2574 scope.go:117] "RemoveContainer" containerID="f51405b52f82c73d104674390284cefdb1278b1310fbf6560f051b7998f8e62f" Jul 19 08:17:04.045601 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:04.045372 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt" Jul 19 08:17:04.066336 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:04.066304 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt"] Jul 19 08:17:04.071555 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:04.071527 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-hpfxt"] Jul 19 08:17:04.731905 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:04.731870 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" path="/var/lib/kubelet/pods/618bb089-02a4-4258-b8ca-69e34763bf1a/volumes" Jul 19 08:17:10.726816 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:10.726772 2574 scope.go:117] "RemoveContainer" containerID="6f2d87544a0d9cfd85e5c40eabf16849d6447f214e7e2d14183035ccf776d199" Jul 19 08:17:11.080027 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:11.079990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" event={"ID":"ac174559-8310-41c4-b2e4-e73e09a392a9","Type":"ContainerStarted","Data":"23d49c2ab00e3e8bb64c629fa4fbfdf625f1c760706ba51043290fec134c1daa"} Jul 19 08:17:12.092506 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:12.092472 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerID="23d49c2ab00e3e8bb64c629fa4fbfdf625f1c760706ba51043290fec134c1daa" exitCode=7 Jul 19 08:17:12.092939 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:12.092541 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" event={"ID":"ac174559-8310-41c4-b2e4-e73e09a392a9","Type":"ContainerDied","Data":"23d49c2ab00e3e8bb64c629fa4fbfdf625f1c760706ba51043290fec134c1daa"} Jul 19 08:17:12.092939 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:12.092644 2574 scope.go:117] "RemoveContainer" containerID="6f2d87544a0d9cfd85e5c40eabf16849d6447f214e7e2d14183035ccf776d199" Jul 19 08:17:12.093057 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:12.092966 2574 scope.go:117] "RemoveContainer" containerID="23d49c2ab00e3e8bb64c629fa4fbfdf625f1c760706ba51043290fec134c1daa" Jul 19 08:17:12.093189 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:12.093165 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd_odh-ai-gateway-infra(ac174559-8310-41c4-b2e4-e73e09a392a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" Jul 19 08:17:12.103261 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:12.103215 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" podStartSLOduration=20.103200653 podStartE2EDuration="20.103200653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:17:11.09382447 +0000 UTC m=+2078.873487785" watchObservedRunningTime="2026-07-19 08:17:12.103200653 +0000 UTC m=+2079.882863966" Jul 19 08:17:12.103425 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:12.103417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd"] Jul 19 08:17:13.224869 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:13.224844 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:17:13.272880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:13.272841 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac174559-8310-41c4-b2e4-e73e09a392a9-maas-api-service-ca\") pod \"ac174559-8310-41c4-b2e4-e73e09a392a9\" (UID: \"ac174559-8310-41c4-b2e4-e73e09a392a9\") " Jul 19 08:17:13.273236 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:13.273211 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac174559-8310-41c4-b2e4-e73e09a392a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac174559-8310-41c4-b2e4-e73e09a392a9" (UID: "ac174559-8310-41c4-b2e4-e73e09a392a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:17:13.374105 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:13.374017 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac174559-8310-41c4-b2e4-e73e09a392a9-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:17:14.103987 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.103956 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" Jul 19 08:17:14.104198 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.103959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd" event={"ID":"ac174559-8310-41c4-b2e4-e73e09a392a9","Type":"ContainerDied","Data":"7015ecc839ea49d263ba0b1d4fa0f5ab3f8ed4324bff50568fd936cb889b76cf"} Jul 19 08:17:14.104198 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.104077 2574 scope.go:117] "RemoveContainer" containerID="23d49c2ab00e3e8bb64c629fa4fbfdf625f1c760706ba51043290fec134c1daa" Jul 19 08:17:14.125213 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.125175 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd"] Jul 19 08:17:14.128686 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.128652 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8kcdd"] Jul 19 08:17:14.564796 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.564703 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6"] Jul 19 08:17:14.565257 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565218 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:14.565257 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565236 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:14.565257 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565254 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565262 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565272 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565281 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565293 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565302 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565403 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:14.565423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565416 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:14.565777 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565427 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:14.565777 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.565439 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:14.569844 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.569823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:14.572343 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.572319 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:17:14.573947 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.573924 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6"] Jul 19 08:17:14.685459 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.685421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee66f6-5e76-4141-925d-8dc734b5fc9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6\" (UID: \"ebee66f6-5e76-4141-925d-8dc734b5fc9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:14.731423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.731387 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" path="/var/lib/kubelet/pods/ac174559-8310-41c4-b2e4-e73e09a392a9/volumes" Jul 19 08:17:14.786654 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.786614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee66f6-5e76-4141-925d-8dc734b5fc9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6\" (UID: \"ebee66f6-5e76-4141-925d-8dc734b5fc9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:14.787299 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.787273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee66f6-5e76-4141-925d-8dc734b5fc9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6\" (UID: \"ebee66f6-5e76-4141-925d-8dc734b5fc9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:14.881861 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:14.881824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:15.016967 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:15.016940 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6"] Jul 19 08:17:15.019639 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:17:15.019609 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebee66f6_5e76_4141_925d_8dc734b5fc9a.slice/crio-80f1a7952d66d29827104755918e7a93d77cd23e66bc7fceea2b18da4b4e311f WatchSource:0}: Error finding container 80f1a7952d66d29827104755918e7a93d77cd23e66bc7fceea2b18da4b4e311f: Status 404 returned error can't find the container with id 80f1a7952d66d29827104755918e7a93d77cd23e66bc7fceea2b18da4b4e311f Jul 19 08:17:15.110465 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:15.110428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" event={"ID":"ebee66f6-5e76-4141-925d-8dc734b5fc9a","Type":"ContainerStarted","Data":"80f1a7952d66d29827104755918e7a93d77cd23e66bc7fceea2b18da4b4e311f"} Jul 19 08:17:16.116033 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:16.116000 2574 generic.go:356] "Generic (PLEG): container finished" podID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerID="1d79336dae1f344585da94e0545a747f6868c814c45a0bc0eac18366a94a0f41" exitCode=6 Jul 19 08:17:16.116406 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:16.116044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" event={"ID":"ebee66f6-5e76-4141-925d-8dc734b5fc9a","Type":"ContainerDied","Data":"1d79336dae1f344585da94e0545a747f6868c814c45a0bc0eac18366a94a0f41"} Jul 19 08:17:16.116406 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:16.116224 2574 scope.go:117] "RemoveContainer" containerID="1d79336dae1f344585da94e0545a747f6868c814c45a0bc0eac18366a94a0f41" Jul 19 08:17:17.122946 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:17.122907 2574 generic.go:356] "Generic (PLEG): container finished" podID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerID="3b833f12859df457feda9421a2e77dbf58529ff2f6111214564cf4f043878b49" exitCode=6 Jul 19 08:17:17.123396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:17.122987 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" event={"ID":"ebee66f6-5e76-4141-925d-8dc734b5fc9a","Type":"ContainerDied","Data":"3b833f12859df457feda9421a2e77dbf58529ff2f6111214564cf4f043878b49"} Jul 19 08:17:17.123396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:17.123067 2574 scope.go:117] "RemoveContainer" containerID="1d79336dae1f344585da94e0545a747f6868c814c45a0bc0eac18366a94a0f41" Jul 19 08:17:17.123396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:17.123292 2574 scope.go:117] "RemoveContainer" containerID="3b833f12859df457feda9421a2e77dbf58529ff2f6111214564cf4f043878b49" Jul 19 08:17:17.123559 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:17.123507 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6_odh-ai-gateway-infra(ebee66f6-5e76-4141-925d-8dc734b5fc9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" Jul 19 08:17:18.130379 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:18.130348 2574 scope.go:117] "RemoveContainer" containerID="3b833f12859df457feda9421a2e77dbf58529ff2f6111214564cf4f043878b49" Jul 19 08:17:18.130779 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:18.130601 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6_odh-ai-gateway-infra(ebee66f6-5e76-4141-925d-8dc734b5fc9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" Jul 19 08:17:31.726237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:31.726203 2574 scope.go:117] "RemoveContainer" containerID="3b833f12859df457feda9421a2e77dbf58529ff2f6111214564cf4f043878b49" Jul 19 08:17:32.202390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:32.202346 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" event={"ID":"ebee66f6-5e76-4141-925d-8dc734b5fc9a","Type":"ContainerStarted","Data":"afc4ebeb7aa0638acc9585ba2e7af9eb96cef6e141fb1dfc5d52d6bfb959ecd8"} Jul 19 08:17:32.217571 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:32.217508 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" podStartSLOduration=18.217487698 podStartE2EDuration="18.217487698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:17:32.216225642 +0000 UTC m=+2099.995888959" watchObservedRunningTime="2026-07-19 08:17:32.217487698 +0000 UTC m=+2099.997151017" Jul 19 08:17:33.208223 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:33.208128 2574 generic.go:356] "Generic (PLEG): container finished" podID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerID="afc4ebeb7aa0638acc9585ba2e7af9eb96cef6e141fb1dfc5d52d6bfb959ecd8" exitCode=6 Jul 19 08:17:33.208223 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:33.208201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" event={"ID":"ebee66f6-5e76-4141-925d-8dc734b5fc9a","Type":"ContainerDied","Data":"afc4ebeb7aa0638acc9585ba2e7af9eb96cef6e141fb1dfc5d52d6bfb959ecd8"} Jul 19 08:17:33.208870 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:33.208258 2574 scope.go:117] "RemoveContainer" containerID="3b833f12859df457feda9421a2e77dbf58529ff2f6111214564cf4f043878b49" Jul 19 08:17:33.208870 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:33.208610 2574 scope.go:117] "RemoveContainer" containerID="afc4ebeb7aa0638acc9585ba2e7af9eb96cef6e141fb1dfc5d52d6bfb959ecd8" Jul 19 08:17:33.208980 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:33.208919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6_odh-ai-gateway-infra(ebee66f6-5e76-4141-925d-8dc734b5fc9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" Jul 19 08:17:33.234573 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:33.234531 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6"] Jul 19 08:17:34.350047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:34.350020 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:34.476974 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:34.476882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee66f6-5e76-4141-925d-8dc734b5fc9a-maas-api-service-ca\") pod \"ebee66f6-5e76-4141-925d-8dc734b5fc9a\" (UID: \"ebee66f6-5e76-4141-925d-8dc734b5fc9a\") " Jul 19 08:17:34.477273 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:34.477247 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebee66f6-5e76-4141-925d-8dc734b5fc9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebee66f6-5e76-4141-925d-8dc734b5fc9a" (UID: "ebee66f6-5e76-4141-925d-8dc734b5fc9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:17:34.578066 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:34.578028 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee66f6-5e76-4141-925d-8dc734b5fc9a-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:17:35.220752 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:35.220716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" Jul 19 08:17:35.220752 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:35.220733 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6" event={"ID":"ebee66f6-5e76-4141-925d-8dc734b5fc9a","Type":"ContainerDied","Data":"80f1a7952d66d29827104755918e7a93d77cd23e66bc7fceea2b18da4b4e311f"} Jul 19 08:17:35.221039 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:35.220798 2574 scope.go:117] "RemoveContainer" containerID="afc4ebeb7aa0638acc9585ba2e7af9eb96cef6e141fb1dfc5d52d6bfb959ecd8" Jul 19 08:17:35.237881 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:35.237852 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6"] Jul 19 08:17:35.246503 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:35.246477 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-4zsz6"] Jul 19 08:17:36.732092 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:36.732050 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" path="/var/lib/kubelet/pods/ebee66f6-5e76-4141-925d-8dc734b5fc9a/volumes" Jul 19 08:17:52.779575 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779533 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4"] Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779904 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779915 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779929 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779934 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779945 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779951 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779961 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779966 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779973 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.779979 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.780034 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="618bb089-02a4-4258-b8ca-69e34763bf1a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.780044 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac174559-8310-41c4-b2e4-e73e09a392a9" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.780049 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.780057 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.780104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.780063 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebee66f6-5e76-4141-925d-8dc734b5fc9a" containerName="revoke-keys" Jul 19 08:17:52.784228 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.784206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:17:52.787903 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.787801 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:17:52.790467 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.790439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4"] Jul 19 08:17:52.847607 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.847568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6dad5c-fef1-470b-9b16-5d944d81f62b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4\" (UID: \"4e6dad5c-fef1-470b-9b16-5d944d81f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:17:52.948951 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.948911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6dad5c-fef1-470b-9b16-5d944d81f62b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4\" (UID: \"4e6dad5c-fef1-470b-9b16-5d944d81f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:17:52.949583 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:52.949560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6dad5c-fef1-470b-9b16-5d944d81f62b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4\" (UID: \"4e6dad5c-fef1-470b-9b16-5d944d81f62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:17:53.100296 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:53.100244 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:17:53.235562 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:53.235492 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4"] Jul 19 08:17:53.238365 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:17:53.238323 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6dad5c_fef1_470b_9b16_5d944d81f62b.slice/crio-cf42c1cf34e9b3d40257e3b14623440f5ffc3781f7e9afc4ccb369568c959ea3 WatchSource:0}: Error finding container cf42c1cf34e9b3d40257e3b14623440f5ffc3781f7e9afc4ccb369568c959ea3: Status 404 returned error can't find the container with id cf42c1cf34e9b3d40257e3b14623440f5ffc3781f7e9afc4ccb369568c959ea3 Jul 19 08:17:53.303488 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:53.303452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" event={"ID":"4e6dad5c-fef1-470b-9b16-5d944d81f62b","Type":"ContainerStarted","Data":"cf42c1cf34e9b3d40257e3b14623440f5ffc3781f7e9afc4ccb369568c959ea3"} Jul 19 08:17:54.310388 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:54.310302 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerID="a5748fa86a59cc64666b34480ea4c62379e1e8c3bc441b766fc3937ef287b4f8" exitCode=7 Jul 19 08:17:54.310388 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:54.310384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" event={"ID":"4e6dad5c-fef1-470b-9b16-5d944d81f62b","Type":"ContainerDied","Data":"a5748fa86a59cc64666b34480ea4c62379e1e8c3bc441b766fc3937ef287b4f8"} Jul 19 08:17:54.310819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:54.310599 2574 scope.go:117] "RemoveContainer" containerID="a5748fa86a59cc64666b34480ea4c62379e1e8c3bc441b766fc3937ef287b4f8" Jul 19 08:17:55.317096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:55.317005 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerID="36f195de51c18b650a627f3abe6f84607b97f7417f61f0f878ca647a14533130" exitCode=7 Jul 19 08:17:55.317463 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:55.317095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" event={"ID":"4e6dad5c-fef1-470b-9b16-5d944d81f62b","Type":"ContainerDied","Data":"36f195de51c18b650a627f3abe6f84607b97f7417f61f0f878ca647a14533130"} Jul 19 08:17:55.317463 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:55.317158 2574 scope.go:117] "RemoveContainer" containerID="a5748fa86a59cc64666b34480ea4c62379e1e8c3bc441b766fc3937ef287b4f8" Jul 19 08:17:55.317463 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:55.317334 2574 scope.go:117] "RemoveContainer" containerID="36f195de51c18b650a627f3abe6f84607b97f7417f61f0f878ca647a14533130" Jul 19 08:17:55.317599 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:55.317553 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4_odh-ai-gateway-infra(4e6dad5c-fef1-470b-9b16-5d944d81f62b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" Jul 19 08:17:56.323096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:17:56.323064 2574 scope.go:117] "RemoveContainer" containerID="36f195de51c18b650a627f3abe6f84607b97f7417f61f0f878ca647a14533130" Jul 19 08:17:56.323456 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:17:56.323244 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4_odh-ai-gateway-infra(4e6dad5c-fef1-470b-9b16-5d944d81f62b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" Jul 19 08:18:10.726486 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:10.726449 2574 scope.go:117] "RemoveContainer" containerID="36f195de51c18b650a627f3abe6f84607b97f7417f61f0f878ca647a14533130" Jul 19 08:18:11.397484 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:11.397453 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerID="c435d00b6a870e60469e5a6ccda181e33861fc6afd2f0c16782a1ff51341830d" exitCode=7 Jul 19 08:18:11.397695 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:11.397534 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" event={"ID":"4e6dad5c-fef1-470b-9b16-5d944d81f62b","Type":"ContainerDied","Data":"c435d00b6a870e60469e5a6ccda181e33861fc6afd2f0c16782a1ff51341830d"} Jul 19 08:18:11.397695 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:11.397600 2574 scope.go:117] "RemoveContainer" containerID="36f195de51c18b650a627f3abe6f84607b97f7417f61f0f878ca647a14533130" Jul 19 08:18:11.397984 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:11.397962 2574 scope.go:117] "RemoveContainer" containerID="c435d00b6a870e60469e5a6ccda181e33861fc6afd2f0c16782a1ff51341830d" Jul 19 08:18:11.398230 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:18:11.398206 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4_odh-ai-gateway-infra(4e6dad5c-fef1-470b-9b16-5d944d81f62b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" Jul 19 08:18:12.423420 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:12.423385 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4"] Jul 19 08:18:12.553112 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:12.553086 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:18:12.738614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:12.738526 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6dad5c-fef1-470b-9b16-5d944d81f62b-maas-api-service-ca\") pod \"4e6dad5c-fef1-470b-9b16-5d944d81f62b\" (UID: \"4e6dad5c-fef1-470b-9b16-5d944d81f62b\") " Jul 19 08:18:12.738966 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:12.738938 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6dad5c-fef1-470b-9b16-5d944d81f62b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e6dad5c-fef1-470b-9b16-5d944d81f62b" (UID: "4e6dad5c-fef1-470b-9b16-5d944d81f62b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:18:12.839933 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:12.839895 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e6dad5c-fef1-470b-9b16-5d944d81f62b-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:18:13.409723 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:13.409661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" event={"ID":"4e6dad5c-fef1-470b-9b16-5d944d81f62b","Type":"ContainerDied","Data":"cf42c1cf34e9b3d40257e3b14623440f5ffc3781f7e9afc4ccb369568c959ea3"} Jul 19 08:18:13.409723 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:13.409717 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4" Jul 19 08:18:13.409970 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:13.409753 2574 scope.go:117] "RemoveContainer" containerID="c435d00b6a870e60469e5a6ccda181e33861fc6afd2f0c16782a1ff51341830d" Jul 19 08:18:13.434026 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:13.433997 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4"] Jul 19 08:18:13.439315 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:13.439288 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-7mwz4"] Jul 19 08:18:14.731391 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:14.731357 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" path="/var/lib/kubelet/pods/4e6dad5c-fef1-470b-9b16-5d944d81f62b/volumes" Jul 19 08:18:52.826757 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.826702 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g"] Jul 19 08:18:52.827315 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827295 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827361 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827321 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827361 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827339 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827361 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827348 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827454 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827361 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827454 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827369 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827454 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827452 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.827549 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.827468 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:18:52.830826 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.830805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:18:52.833103 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.833069 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:18:52.836196 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.836164 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g"] Jul 19 08:18:52.993025 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:52.992991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38b998f-7817-4696-a167-1ae7b6b07c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g\" (UID: \"a38b998f-7817-4696-a167-1ae7b6b07c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:18:53.093764 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.093726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38b998f-7817-4696-a167-1ae7b6b07c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g\" (UID: \"a38b998f-7817-4696-a167-1ae7b6b07c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:18:53.094492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.094471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38b998f-7817-4696-a167-1ae7b6b07c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g\" (UID: \"a38b998f-7817-4696-a167-1ae7b6b07c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:18:53.143625 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.143589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:18:53.284092 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.283955 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g"] Jul 19 08:18:53.286694 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:18:53.286657 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38b998f_7817_4696_a167_1ae7b6b07c16.slice/crio-19d9ef73d4ca9619fc480a2a94fcf166a1753896ccb031a8418dfb49e9386d72 WatchSource:0}: Error finding container 19d9ef73d4ca9619fc480a2a94fcf166a1753896ccb031a8418dfb49e9386d72: Status 404 returned error can't find the container with id 19d9ef73d4ca9619fc480a2a94fcf166a1753896ccb031a8418dfb49e9386d72 Jul 19 08:18:53.612016 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.611985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" event={"ID":"a38b998f-7817-4696-a167-1ae7b6b07c16","Type":"ContainerStarted","Data":"6895bd9ea99e9a6ae7079b44004f515a51de00586761cc97aa3b22c7b6000488"} Jul 19 08:18:53.612162 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.612036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" event={"ID":"a38b998f-7817-4696-a167-1ae7b6b07c16","Type":"ContainerStarted","Data":"19d9ef73d4ca9619fc480a2a94fcf166a1753896ccb031a8418dfb49e9386d72"} Jul 19 08:18:53.626444 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:53.626389 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" podStartSLOduration=1.626374126 podStartE2EDuration="1.626374126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:18:53.624703912 +0000 UTC m=+2181.404367245" watchObservedRunningTime="2026-07-19 08:18:53.626374126 +0000 UTC m=+2181.406037439" Jul 19 08:18:54.618425 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:54.618387 2574 generic.go:356] "Generic (PLEG): container finished" podID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerID="6895bd9ea99e9a6ae7079b44004f515a51de00586761cc97aa3b22c7b6000488" exitCode=7 Jul 19 08:18:54.618844 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:54.618473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" event={"ID":"a38b998f-7817-4696-a167-1ae7b6b07c16","Type":"ContainerDied","Data":"6895bd9ea99e9a6ae7079b44004f515a51de00586761cc97aa3b22c7b6000488"} Jul 19 08:18:54.618844 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:54.618653 2574 scope.go:117] "RemoveContainer" containerID="6895bd9ea99e9a6ae7079b44004f515a51de00586761cc97aa3b22c7b6000488" Jul 19 08:18:55.625595 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:55.625553 2574 generic.go:356] "Generic (PLEG): container finished" podID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerID="900dfc0f6210a9671a5004e93f075247cd68ea128ae3cb3e8cce4e3f7f10efae" exitCode=7 Jul 19 08:18:55.626069 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:55.625615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" event={"ID":"a38b998f-7817-4696-a167-1ae7b6b07c16","Type":"ContainerDied","Data":"900dfc0f6210a9671a5004e93f075247cd68ea128ae3cb3e8cce4e3f7f10efae"} Jul 19 08:18:55.626069 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:55.625705 2574 scope.go:117] "RemoveContainer" containerID="6895bd9ea99e9a6ae7079b44004f515a51de00586761cc97aa3b22c7b6000488" Jul 19 08:18:55.626069 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:55.625911 2574 scope.go:117] "RemoveContainer" containerID="900dfc0f6210a9671a5004e93f075247cd68ea128ae3cb3e8cce4e3f7f10efae" Jul 19 08:18:55.626249 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:18:55.626131 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g_odh-ai-gateway-infra(a38b998f-7817-4696-a167-1ae7b6b07c16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" Jul 19 08:18:56.632129 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:18:56.632099 2574 scope.go:117] "RemoveContainer" containerID="900dfc0f6210a9671a5004e93f075247cd68ea128ae3cb3e8cce4e3f7f10efae" Jul 19 08:18:56.632550 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:18:56.632304 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g_odh-ai-gateway-infra(a38b998f-7817-4696-a167-1ae7b6b07c16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" Jul 19 08:19:07.726805 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:07.726768 2574 scope.go:117] "RemoveContainer" containerID="900dfc0f6210a9671a5004e93f075247cd68ea128ae3cb3e8cce4e3f7f10efae" Jul 19 08:19:08.689088 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:08.689056 2574 generic.go:356] "Generic (PLEG): container finished" podID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerID="178e011ee551d30c78ee2e518b50ed376c522e089fbd2dd8520bf3c11130b74c" exitCode=7 Jul 19 08:19:08.689301 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:08.689130 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" event={"ID":"a38b998f-7817-4696-a167-1ae7b6b07c16","Type":"ContainerDied","Data":"178e011ee551d30c78ee2e518b50ed376c522e089fbd2dd8520bf3c11130b74c"} Jul 19 08:19:08.689301 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:08.689198 2574 scope.go:117] "RemoveContainer" containerID="900dfc0f6210a9671a5004e93f075247cd68ea128ae3cb3e8cce4e3f7f10efae" Jul 19 08:19:08.689565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:08.689539 2574 scope.go:117] "RemoveContainer" containerID="178e011ee551d30c78ee2e518b50ed376c522e089fbd2dd8520bf3c11130b74c" Jul 19 08:19:08.689904 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:19:08.689868 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g_odh-ai-gateway-infra(a38b998f-7817-4696-a167-1ae7b6b07c16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" Jul 19 08:19:09.712555 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:09.712518 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g"] Jul 19 08:19:09.847621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:09.847595 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:19:09.947187 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:09.947153 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38b998f-7817-4696-a167-1ae7b6b07c16-maas-api-service-ca\") pod \"a38b998f-7817-4696-a167-1ae7b6b07c16\" (UID: \"a38b998f-7817-4696-a167-1ae7b6b07c16\") " Jul 19 08:19:09.947543 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:09.947520 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38b998f-7817-4696-a167-1ae7b6b07c16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a38b998f-7817-4696-a167-1ae7b6b07c16" (UID: "a38b998f-7817-4696-a167-1ae7b6b07c16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:19:10.048358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:10.048268 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38b998f-7817-4696-a167-1ae7b6b07c16-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:19:10.701070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:10.701043 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" Jul 19 08:19:10.701253 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:10.701040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g" event={"ID":"a38b998f-7817-4696-a167-1ae7b6b07c16","Type":"ContainerDied","Data":"19d9ef73d4ca9619fc480a2a94fcf166a1753896ccb031a8418dfb49e9386d72"} Jul 19 08:19:10.701253 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:10.701174 2574 scope.go:117] "RemoveContainer" containerID="178e011ee551d30c78ee2e518b50ed376c522e089fbd2dd8520bf3c11130b74c" Jul 19 08:19:10.723583 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:10.723557 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g"] Jul 19 08:19:10.732308 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:10.732283 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-hw69g"] Jul 19 08:19:12.732234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:12.732199 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" path="/var/lib/kubelet/pods/a38b998f-7817-4696-a167-1ae7b6b07c16/volumes" Jul 19 08:19:42.863645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.863612 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd"] Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864007 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864019 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864028 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864034 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864106 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6dad5c-fef1-470b-9b16-5d944d81f62b" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864118 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864124 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.864138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.864129 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:19:42.867168 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.867151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:19:42.869377 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.869356 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:19:42.873588 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:42.873563 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd"] Jul 19 08:19:43.048063 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.048021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d545eb99-7d5e-4f76-a582-d72feeac3fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd\" (UID: \"d545eb99-7d5e-4f76-a582-d72feeac3fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:19:43.149222 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.149126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d545eb99-7d5e-4f76-a582-d72feeac3fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd\" (UID: \"d545eb99-7d5e-4f76-a582-d72feeac3fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:19:43.149847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.149824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d545eb99-7d5e-4f76-a582-d72feeac3fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd\" (UID: \"d545eb99-7d5e-4f76-a582-d72feeac3fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:19:43.179692 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.179652 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:19:43.322187 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.322163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd"] Jul 19 08:19:43.324994 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:19:43.324967 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd545eb99_7d5e_4f76_a582_d72feeac3fe6.slice/crio-b61cc8f095e98b6a91f062596f2e9ec8a2f1488a60197d3ab1696c3f5534f0fb WatchSource:0}: Error finding container b61cc8f095e98b6a91f062596f2e9ec8a2f1488a60197d3ab1696c3f5534f0fb: Status 404 returned error can't find the container with id b61cc8f095e98b6a91f062596f2e9ec8a2f1488a60197d3ab1696c3f5534f0fb Jul 19 08:19:43.851521 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.851480 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerStarted","Data":"5a2403fa578091d96e8081ede3275fd2ad0e8d0c80db39153e73c89108894966"} Jul 19 08:19:43.851752 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.851533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerStarted","Data":"b61cc8f095e98b6a91f062596f2e9ec8a2f1488a60197d3ab1696c3f5534f0fb"} Jul 19 08:19:43.865327 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:43.865272 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" podStartSLOduration=1.865225658 podStartE2EDuration="1.865225658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:19:43.864337824 +0000 UTC m=+2231.644001138" watchObservedRunningTime="2026-07-19 08:19:43.865225658 +0000 UTC m=+2231.644888973" Jul 19 08:19:44.857781 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:44.857746 2574 generic.go:356] "Generic (PLEG): container finished" podID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerID="5a2403fa578091d96e8081ede3275fd2ad0e8d0c80db39153e73c89108894966" exitCode=7 Jul 19 08:19:44.857975 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:44.857822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerDied","Data":"5a2403fa578091d96e8081ede3275fd2ad0e8d0c80db39153e73c89108894966"} Jul 19 08:19:44.858088 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:44.858075 2574 scope.go:117] "RemoveContainer" containerID="5a2403fa578091d96e8081ede3275fd2ad0e8d0c80db39153e73c89108894966" Jul 19 08:19:45.864175 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:45.864140 2574 generic.go:356] "Generic (PLEG): container finished" podID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerID="8c70f19186b62fb848804b3def6aacb5d4704dd58a70e1c90df660a8f378cc8b" exitCode=7 Jul 19 08:19:45.864656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:45.864234 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerDied","Data":"8c70f19186b62fb848804b3def6aacb5d4704dd58a70e1c90df660a8f378cc8b"} Jul 19 08:19:45.864656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:45.864300 2574 scope.go:117] "RemoveContainer" containerID="5a2403fa578091d96e8081ede3275fd2ad0e8d0c80db39153e73c89108894966" Jul 19 08:19:45.864656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:45.864591 2574 scope.go:117] "RemoveContainer" containerID="8c70f19186b62fb848804b3def6aacb5d4704dd58a70e1c90df660a8f378cc8b" Jul 19 08:19:45.864867 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:19:45.864841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd_odh-ai-gateway-infra(d545eb99-7d5e-4f76-a582-d72feeac3fe6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" Jul 19 08:19:46.871012 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:46.870983 2574 scope.go:117] "RemoveContainer" containerID="8c70f19186b62fb848804b3def6aacb5d4704dd58a70e1c90df660a8f378cc8b" Jul 19 08:19:46.871389 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:19:46.871172 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd_odh-ai-gateway-infra(d545eb99-7d5e-4f76-a582-d72feeac3fe6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" Jul 19 08:19:57.726250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:57.726159 2574 scope.go:117] "RemoveContainer" containerID="8c70f19186b62fb848804b3def6aacb5d4704dd58a70e1c90df660a8f378cc8b" Jul 19 08:19:57.928126 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:57.928085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerStarted","Data":"1acdd4f6063bee3e565b179a8304cd471d0bcec844d3c0a1e576e2514a48de25"} Jul 19 08:19:58.933823 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:58.933791 2574 generic.go:356] "Generic (PLEG): container finished" podID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerID="1acdd4f6063bee3e565b179a8304cd471d0bcec844d3c0a1e576e2514a48de25" exitCode=7 Jul 19 08:19:58.934229 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:58.933865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerDied","Data":"1acdd4f6063bee3e565b179a8304cd471d0bcec844d3c0a1e576e2514a48de25"} Jul 19 08:19:58.934229 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:58.933927 2574 scope.go:117] "RemoveContainer" containerID="8c70f19186b62fb848804b3def6aacb5d4704dd58a70e1c90df660a8f378cc8b" Jul 19 08:19:58.934229 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:58.934144 2574 scope.go:117] "RemoveContainer" containerID="1acdd4f6063bee3e565b179a8304cd471d0bcec844d3c0a1e576e2514a48de25" Jul 19 08:19:58.934401 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:19:58.934363 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd_odh-ai-gateway-infra(d545eb99-7d5e-4f76-a582-d72feeac3fe6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" Jul 19 08:19:58.956559 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:19:58.956533 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd"] Jul 19 08:20:00.078165 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.078139 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:20:00.195096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.194999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d545eb99-7d5e-4f76-a582-d72feeac3fe6-maas-api-service-ca\") pod \"d545eb99-7d5e-4f76-a582-d72feeac3fe6\" (UID: \"d545eb99-7d5e-4f76-a582-d72feeac3fe6\") " Jul 19 08:20:00.195372 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.195347 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d545eb99-7d5e-4f76-a582-d72feeac3fe6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d545eb99-7d5e-4f76-a582-d72feeac3fe6" (UID: "d545eb99-7d5e-4f76-a582-d72feeac3fe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:20:00.296511 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.296463 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d545eb99-7d5e-4f76-a582-d72feeac3fe6-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:20:00.945066 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.945031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" event={"ID":"d545eb99-7d5e-4f76-a582-d72feeac3fe6","Type":"ContainerDied","Data":"b61cc8f095e98b6a91f062596f2e9ec8a2f1488a60197d3ab1696c3f5534f0fb"} Jul 19 08:20:00.945066 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.945066 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd" Jul 19 08:20:00.945322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.945095 2574 scope.go:117] "RemoveContainer" containerID="1acdd4f6063bee3e565b179a8304cd471d0bcec844d3c0a1e576e2514a48de25" Jul 19 08:20:00.959225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.959194 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd"] Jul 19 08:20:00.961270 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:00.961247 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-t2xvd"] Jul 19 08:20:02.737974 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:02.737933 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" path="/var/lib/kubelet/pods/d545eb99-7d5e-4f76-a582-d72feeac3fe6/volumes" Jul 19 08:20:04.703450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703417 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf"] Jul 19 08:20:04.703847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703808 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.703847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703820 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.703847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703830 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.703847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703836 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.703847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703848 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.704008 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703853 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.704008 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703868 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:20:04.704008 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703877 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38b998f-7817-4696-a167-1ae7b6b07c16" containerName="revoke-keys" Jul 19 08:20:04.704008 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703940 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.704008 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703949 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.704008 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.703960 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d545eb99-7d5e-4f76-a582-d72feeac3fe6" containerName="revoke-keys" Jul 19 08:20:04.708503 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.708485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:04.711842 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.711712 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:20:04.712859 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.712791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf"] Jul 19 08:20:04.740346 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.740306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b710b-31bd-44cb-b99a-04838cdc8fdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf\" (UID: \"209b710b-31bd-44cb-b99a-04838cdc8fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:04.841473 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.841432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b710b-31bd-44cb-b99a-04838cdc8fdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf\" (UID: \"209b710b-31bd-44cb-b99a-04838cdc8fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:04.842092 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:04.842072 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b710b-31bd-44cb-b99a-04838cdc8fdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf\" (UID: \"209b710b-31bd-44cb-b99a-04838cdc8fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:05.020763 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:05.020638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:05.153047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:05.153024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf"] Jul 19 08:20:05.155239 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:20:05.155209 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209b710b_31bd_44cb_b99a_04838cdc8fdc.slice/crio-2b7a24b28950a1dabd6a13942c2c9c139b7af403f7f389e27f5f11d55ced7a04 WatchSource:0}: Error finding container 2b7a24b28950a1dabd6a13942c2c9c139b7af403f7f389e27f5f11d55ced7a04: Status 404 returned error can't find the container with id 2b7a24b28950a1dabd6a13942c2c9c139b7af403f7f389e27f5f11d55ced7a04 Jul 19 08:20:05.969432 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:05.969392 2574 generic.go:356] "Generic (PLEG): container finished" podID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerID="18f7dcf76d909afb829238f5ee59720c53e6b30f9e1103b81c8684e1fa30a7e0" exitCode=6 Jul 19 08:20:05.969432 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:05.969433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" event={"ID":"209b710b-31bd-44cb-b99a-04838cdc8fdc","Type":"ContainerDied","Data":"18f7dcf76d909afb829238f5ee59720c53e6b30f9e1103b81c8684e1fa30a7e0"} Jul 19 08:20:05.969891 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:05.969470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" event={"ID":"209b710b-31bd-44cb-b99a-04838cdc8fdc","Type":"ContainerStarted","Data":"2b7a24b28950a1dabd6a13942c2c9c139b7af403f7f389e27f5f11d55ced7a04"} Jul 19 08:20:05.969891 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:05.969649 2574 scope.go:117] "RemoveContainer" containerID="18f7dcf76d909afb829238f5ee59720c53e6b30f9e1103b81c8684e1fa30a7e0" Jul 19 08:20:06.982012 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:06.981970 2574 generic.go:356] "Generic (PLEG): container finished" podID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerID="f47cc5b81e6cd237ab82ec0708b87b1fa2e04ce003941016c16b5d9841fc9d74" exitCode=6 Jul 19 08:20:06.982492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:06.982060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" event={"ID":"209b710b-31bd-44cb-b99a-04838cdc8fdc","Type":"ContainerDied","Data":"f47cc5b81e6cd237ab82ec0708b87b1fa2e04ce003941016c16b5d9841fc9d74"} Jul 19 08:20:06.982492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:06.982142 2574 scope.go:117] "RemoveContainer" containerID="18f7dcf76d909afb829238f5ee59720c53e6b30f9e1103b81c8684e1fa30a7e0" Jul 19 08:20:06.982492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:06.982222 2574 scope.go:117] "RemoveContainer" containerID="f47cc5b81e6cd237ab82ec0708b87b1fa2e04ce003941016c16b5d9841fc9d74" Jul 19 08:20:06.982492 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:06.982447 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf_odh-ai-gateway-infra(209b710b-31bd-44cb-b99a-04838cdc8fdc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" Jul 19 08:20:07.988672 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:07.988641 2574 scope.go:117] "RemoveContainer" containerID="f47cc5b81e6cd237ab82ec0708b87b1fa2e04ce003941016c16b5d9841fc9d74" Jul 19 08:20:07.989069 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:07.988856 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf_odh-ai-gateway-infra(209b710b-31bd-44cb-b99a-04838cdc8fdc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" Jul 19 08:20:19.726742 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:19.726672 2574 scope.go:117] "RemoveContainer" containerID="f47cc5b81e6cd237ab82ec0708b87b1fa2e04ce003941016c16b5d9841fc9d74" Jul 19 08:20:20.044218 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:20.044133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" event={"ID":"209b710b-31bd-44cb-b99a-04838cdc8fdc","Type":"ContainerStarted","Data":"8db0ee3065dc060f55eeddfd50317deea842b5efda302b58c33a503cc7283826"} Jul 19 08:20:20.067630 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:20.067562 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" podStartSLOduration=16.067539619 podStartE2EDuration="16.067539619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:20:20.058242773 +0000 UTC m=+2267.837906079" watchObservedRunningTime="2026-07-19 08:20:20.067539619 +0000 UTC m=+2267.847202933" Jul 19 08:20:21.050438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:21.050399 2574 generic.go:356] "Generic (PLEG): container finished" podID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerID="8db0ee3065dc060f55eeddfd50317deea842b5efda302b58c33a503cc7283826" exitCode=6 Jul 19 08:20:21.052884 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:21.050477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" event={"ID":"209b710b-31bd-44cb-b99a-04838cdc8fdc","Type":"ContainerDied","Data":"8db0ee3065dc060f55eeddfd50317deea842b5efda302b58c33a503cc7283826"} Jul 19 08:20:21.052884 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:21.050546 2574 scope.go:117] "RemoveContainer" containerID="f47cc5b81e6cd237ab82ec0708b87b1fa2e04ce003941016c16b5d9841fc9d74" Jul 19 08:20:21.052884 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:21.050853 2574 scope.go:117] "RemoveContainer" containerID="8db0ee3065dc060f55eeddfd50317deea842b5efda302b58c33a503cc7283826" Jul 19 08:20:21.052884 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:21.051081 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf_odh-ai-gateway-infra(209b710b-31bd-44cb-b99a-04838cdc8fdc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" Jul 19 08:20:21.070437 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:21.070406 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf"] Jul 19 08:20:22.195019 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:22.194991 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:22.310297 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:22.310207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b710b-31bd-44cb-b99a-04838cdc8fdc-maas-api-service-ca\") pod \"209b710b-31bd-44cb-b99a-04838cdc8fdc\" (UID: \"209b710b-31bd-44cb-b99a-04838cdc8fdc\") " Jul 19 08:20:22.310566 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:22.310541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209b710b-31bd-44cb-b99a-04838cdc8fdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "209b710b-31bd-44cb-b99a-04838cdc8fdc" (UID: "209b710b-31bd-44cb-b99a-04838cdc8fdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:20:22.411385 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:22.411353 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/209b710b-31bd-44cb-b99a-04838cdc8fdc-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:20:23.062564 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:23.062482 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" Jul 19 08:20:23.062564 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:23.062498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf" event={"ID":"209b710b-31bd-44cb-b99a-04838cdc8fdc","Type":"ContainerDied","Data":"2b7a24b28950a1dabd6a13942c2c9c139b7af403f7f389e27f5f11d55ced7a04"} Jul 19 08:20:23.062815 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:23.062566 2574 scope.go:117] "RemoveContainer" containerID="8db0ee3065dc060f55eeddfd50317deea842b5efda302b58c33a503cc7283826" Jul 19 08:20:23.077747 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:23.077720 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf"] Jul 19 08:20:23.082531 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:23.082508 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wl7xf"] Jul 19 08:20:24.732349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:24.732319 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" path="/var/lib/kubelet/pods/209b710b-31bd-44cb-b99a-04838cdc8fdc/volumes" Jul 19 08:20:32.909160 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909136 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq"] Jul 19 08:20:32.909514 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909485 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:32.909514 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909496 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:32.909514 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909507 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:32.909514 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909513 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:32.909752 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909602 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:32.909752 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.909612 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:32.913963 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.913942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:32.916871 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.916846 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:20:32.919596 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:32.919571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq"] Jul 19 08:20:33.007107 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:33.007067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578a50a-a6d6-4887-b62c-f254479d0ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq\" (UID: \"e578a50a-a6d6-4887-b62c-f254479d0ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:33.108262 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:33.108230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578a50a-a6d6-4887-b62c-f254479d0ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq\" (UID: \"e578a50a-a6d6-4887-b62c-f254479d0ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:33.109054 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:33.109032 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578a50a-a6d6-4887-b62c-f254479d0ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq\" (UID: \"e578a50a-a6d6-4887-b62c-f254479d0ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:33.226596 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:33.226507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:33.364512 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:33.364485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq"] Jul 19 08:20:33.367306 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:20:33.367277 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode578a50a_a6d6_4887_b62c_f254479d0ba4.slice/crio-901faf89687f7c54941b28b05e403af129dea3df5c35392fe3c074dc03d4a787 WatchSource:0}: Error finding container 901faf89687f7c54941b28b05e403af129dea3df5c35392fe3c074dc03d4a787: Status 404 returned error can't find the container with id 901faf89687f7c54941b28b05e403af129dea3df5c35392fe3c074dc03d4a787 Jul 19 08:20:34.115633 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:34.115595 2574 generic.go:356] "Generic (PLEG): container finished" podID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerID="212b5f2402a2a801f12c0dd71821e8cc2c271c698e09f23b1983fd9e1059527c" exitCode=7 Jul 19 08:20:34.116064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:34.115701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" event={"ID":"e578a50a-a6d6-4887-b62c-f254479d0ba4","Type":"ContainerDied","Data":"212b5f2402a2a801f12c0dd71821e8cc2c271c698e09f23b1983fd9e1059527c"} Jul 19 08:20:34.116064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:34.115743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" event={"ID":"e578a50a-a6d6-4887-b62c-f254479d0ba4","Type":"ContainerStarted","Data":"901faf89687f7c54941b28b05e403af129dea3df5c35392fe3c074dc03d4a787"} Jul 19 08:20:34.116064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:34.115861 2574 scope.go:117] "RemoveContainer" containerID="212b5f2402a2a801f12c0dd71821e8cc2c271c698e09f23b1983fd9e1059527c" Jul 19 08:20:35.122234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:35.122197 2574 generic.go:356] "Generic (PLEG): container finished" podID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerID="edcd29ba480cac74cabe68b2f4dd9ed7f461993da2a25be800a4a534bb808b93" exitCode=7 Jul 19 08:20:35.122624 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:35.122279 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" event={"ID":"e578a50a-a6d6-4887-b62c-f254479d0ba4","Type":"ContainerDied","Data":"edcd29ba480cac74cabe68b2f4dd9ed7f461993da2a25be800a4a534bb808b93"} Jul 19 08:20:35.122624 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:35.122333 2574 scope.go:117] "RemoveContainer" containerID="212b5f2402a2a801f12c0dd71821e8cc2c271c698e09f23b1983fd9e1059527c" Jul 19 08:20:35.122624 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:35.122563 2574 scope.go:117] "RemoveContainer" containerID="edcd29ba480cac74cabe68b2f4dd9ed7f461993da2a25be800a4a534bb808b93" Jul 19 08:20:35.122867 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:35.122843 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq_odh-ai-gateway-infra(e578a50a-a6d6-4887-b62c-f254479d0ba4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" Jul 19 08:20:36.129743 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:36.129714 2574 scope.go:117] "RemoveContainer" containerID="edcd29ba480cac74cabe68b2f4dd9ed7f461993da2a25be800a4a534bb808b93" Jul 19 08:20:36.130201 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:36.129971 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq_odh-ai-gateway-infra(e578a50a-a6d6-4887-b62c-f254479d0ba4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" Jul 19 08:20:37.021042 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.021006 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4"] Jul 19 08:20:37.021399 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.021387 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:37.021446 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.021401 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:37.021487 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.021478 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="209b710b-31bd-44cb-b99a-04838cdc8fdc" containerName="revoke-keys" Jul 19 08:20:37.024370 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.024355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:37.031020 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.030995 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4"] Jul 19 08:20:37.046497 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.046460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c680f90b-e292-4c1d-85df-d0029c9bfa80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4\" (UID: \"c680f90b-e292-4c1d-85df-d0029c9bfa80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:37.147167 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.147126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c680f90b-e292-4c1d-85df-d0029c9bfa80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4\" (UID: \"c680f90b-e292-4c1d-85df-d0029c9bfa80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:37.147818 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.147797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c680f90b-e292-4c1d-85df-d0029c9bfa80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4\" (UID: \"c680f90b-e292-4c1d-85df-d0029c9bfa80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:37.337478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.337450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:37.474959 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:37.474928 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4"] Jul 19 08:20:37.477108 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:20:37.477078 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc680f90b_e292_4c1d_85df_d0029c9bfa80.slice/crio-7108c807e0aae22e11cb721c3cf807a109e00854666616f95f61eebb920bc4ff WatchSource:0}: Error finding container 7108c807e0aae22e11cb721c3cf807a109e00854666616f95f61eebb920bc4ff: Status 404 returned error can't find the container with id 7108c807e0aae22e11cb721c3cf807a109e00854666616f95f61eebb920bc4ff Jul 19 08:20:38.140365 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:38.140330 2574 generic.go:356] "Generic (PLEG): container finished" podID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerID="2136f044483314548e1028c99daef854203dbf9374d059c04e026dcffec76972" exitCode=6 Jul 19 08:20:38.140547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:38.140421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" event={"ID":"c680f90b-e292-4c1d-85df-d0029c9bfa80","Type":"ContainerDied","Data":"2136f044483314548e1028c99daef854203dbf9374d059c04e026dcffec76972"} Jul 19 08:20:38.140547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:38.140476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" event={"ID":"c680f90b-e292-4c1d-85df-d0029c9bfa80","Type":"ContainerStarted","Data":"7108c807e0aae22e11cb721c3cf807a109e00854666616f95f61eebb920bc4ff"} Jul 19 08:20:38.140717 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:38.140623 2574 scope.go:117] "RemoveContainer" containerID="2136f044483314548e1028c99daef854203dbf9374d059c04e026dcffec76972" Jul 19 08:20:39.146506 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:39.146471 2574 generic.go:356] "Generic (PLEG): container finished" podID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerID="8495b11f1b5b0e9804943eeeee723df15296985296ae74d20d202547f5c93431" exitCode=6 Jul 19 08:20:39.147027 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:39.146549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" event={"ID":"c680f90b-e292-4c1d-85df-d0029c9bfa80","Type":"ContainerDied","Data":"8495b11f1b5b0e9804943eeeee723df15296985296ae74d20d202547f5c93431"} Jul 19 08:20:39.147027 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:39.146599 2574 scope.go:117] "RemoveContainer" containerID="2136f044483314548e1028c99daef854203dbf9374d059c04e026dcffec76972" Jul 19 08:20:39.147027 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:39.146840 2574 scope.go:117] "RemoveContainer" containerID="8495b11f1b5b0e9804943eeeee723df15296985296ae74d20d202547f5c93431" Jul 19 08:20:39.147190 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:39.147090 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4_odh-ai-gateway-infra(c680f90b-e292-4c1d-85df-d0029c9bfa80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" Jul 19 08:20:40.152338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:40.152310 2574 scope.go:117] "RemoveContainer" containerID="8495b11f1b5b0e9804943eeeee723df15296985296ae74d20d202547f5c93431" Jul 19 08:20:40.152767 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:40.152507 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4_odh-ai-gateway-infra(c680f90b-e292-4c1d-85df-d0029c9bfa80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" Jul 19 08:20:51.726166 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:51.726132 2574 scope.go:117] "RemoveContainer" containerID="edcd29ba480cac74cabe68b2f4dd9ed7f461993da2a25be800a4a534bb808b93" Jul 19 08:20:52.211774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:52.211733 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" event={"ID":"e578a50a-a6d6-4887-b62c-f254479d0ba4","Type":"ContainerStarted","Data":"f5881cee0289714e92de8d4308f35cefebe562402e3a1770f78c38e5333ea9b0"} Jul 19 08:20:52.227003 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:52.226952 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" podStartSLOduration=20.226936145 podStartE2EDuration="20.226936145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:20:52.225156554 +0000 UTC m=+2300.004819868" watchObservedRunningTime="2026-07-19 08:20:52.226936145 +0000 UTC m=+2300.006599458" Jul 19 08:20:52.730048 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:52.730019 2574 scope.go:117] "RemoveContainer" containerID="8495b11f1b5b0e9804943eeeee723df15296985296ae74d20d202547f5c93431" Jul 19 08:20:53.217724 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.217691 2574 generic.go:356] "Generic (PLEG): container finished" podID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerID="f5881cee0289714e92de8d4308f35cefebe562402e3a1770f78c38e5333ea9b0" exitCode=7 Jul 19 08:20:53.217916 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.217750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" event={"ID":"e578a50a-a6d6-4887-b62c-f254479d0ba4","Type":"ContainerDied","Data":"f5881cee0289714e92de8d4308f35cefebe562402e3a1770f78c38e5333ea9b0"} Jul 19 08:20:53.217916 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.217808 2574 scope.go:117] "RemoveContainer" containerID="edcd29ba480cac74cabe68b2f4dd9ed7f461993da2a25be800a4a534bb808b93" Jul 19 08:20:53.218135 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.218112 2574 scope.go:117] "RemoveContainer" containerID="f5881cee0289714e92de8d4308f35cefebe562402e3a1770f78c38e5333ea9b0" Jul 19 08:20:53.218385 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:53.218360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq_odh-ai-gateway-infra(e578a50a-a6d6-4887-b62c-f254479d0ba4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" Jul 19 08:20:53.222456 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.222428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" event={"ID":"c680f90b-e292-4c1d-85df-d0029c9bfa80","Type":"ContainerStarted","Data":"9cf3fd9e4e78f4f1dc7faa6b3e808e00ccb6f0dfbf07d894b473542cd908b272"} Jul 19 08:20:53.240313 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.240284 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq"] Jul 19 08:20:53.249453 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:53.249398 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" podStartSLOduration=16.249382326 podStartE2EDuration="16.249382326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:20:53.247859191 +0000 UTC m=+2301.027522507" watchObservedRunningTime="2026-07-19 08:20:53.249382326 +0000 UTC m=+2301.029045640" Jul 19 08:20:54.235601 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.235563 2574 generic.go:356] "Generic (PLEG): container finished" podID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerID="9cf3fd9e4e78f4f1dc7faa6b3e808e00ccb6f0dfbf07d894b473542cd908b272" exitCode=6 Jul 19 08:20:54.236108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.235637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" event={"ID":"c680f90b-e292-4c1d-85df-d0029c9bfa80","Type":"ContainerDied","Data":"9cf3fd9e4e78f4f1dc7faa6b3e808e00ccb6f0dfbf07d894b473542cd908b272"} Jul 19 08:20:54.236108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.235745 2574 scope.go:117] "RemoveContainer" containerID="8495b11f1b5b0e9804943eeeee723df15296985296ae74d20d202547f5c93431" Jul 19 08:20:54.236108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.236060 2574 scope.go:117] "RemoveContainer" containerID="9cf3fd9e4e78f4f1dc7faa6b3e808e00ccb6f0dfbf07d894b473542cd908b272" Jul 19 08:20:54.236324 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:20:54.236301 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4_odh-ai-gateway-infra(c680f90b-e292-4c1d-85df-d0029c9bfa80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" Jul 19 08:20:54.257294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.257266 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4"] Jul 19 08:20:54.380264 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.380235 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:54.514464 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.514358 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578a50a-a6d6-4887-b62c-f254479d0ba4-maas-api-service-ca\") pod \"e578a50a-a6d6-4887-b62c-f254479d0ba4\" (UID: \"e578a50a-a6d6-4887-b62c-f254479d0ba4\") " Jul 19 08:20:54.514769 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.514739 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e578a50a-a6d6-4887-b62c-f254479d0ba4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e578a50a-a6d6-4887-b62c-f254479d0ba4" (UID: "e578a50a-a6d6-4887-b62c-f254479d0ba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:20:54.615605 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:54.615573 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578a50a-a6d6-4887-b62c-f254479d0ba4-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:20:55.244513 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.244490 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" Jul 19 08:20:55.244851 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.244533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq" event={"ID":"e578a50a-a6d6-4887-b62c-f254479d0ba4","Type":"ContainerDied","Data":"901faf89687f7c54941b28b05e403af129dea3df5c35392fe3c074dc03d4a787"} Jul 19 08:20:55.244851 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.244594 2574 scope.go:117] "RemoveContainer" containerID="f5881cee0289714e92de8d4308f35cefebe562402e3a1770f78c38e5333ea9b0" Jul 19 08:20:55.270808 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.270777 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq"] Jul 19 08:20:55.275308 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.275277 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-8lllq"] Jul 19 08:20:55.386539 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.386516 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:55.524207 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.524108 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c680f90b-e292-4c1d-85df-d0029c9bfa80-maas-api-service-ca\") pod \"c680f90b-e292-4c1d-85df-d0029c9bfa80\" (UID: \"c680f90b-e292-4c1d-85df-d0029c9bfa80\") " Jul 19 08:20:55.524479 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.524455 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c680f90b-e292-4c1d-85df-d0029c9bfa80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c680f90b-e292-4c1d-85df-d0029c9bfa80" (UID: "c680f90b-e292-4c1d-85df-d0029c9bfa80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:20:55.624827 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:55.624794 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c680f90b-e292-4c1d-85df-d0029c9bfa80-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:20:56.250286 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.250226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" event={"ID":"c680f90b-e292-4c1d-85df-d0029c9bfa80","Type":"ContainerDied","Data":"7108c807e0aae22e11cb721c3cf807a109e00854666616f95f61eebb920bc4ff"} Jul 19 08:20:56.250906 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.250350 2574 scope.go:117] "RemoveContainer" containerID="9cf3fd9e4e78f4f1dc7faa6b3e808e00ccb6f0dfbf07d894b473542cd908b272" Jul 19 08:20:56.250906 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.250259 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4" Jul 19 08:20:56.272360 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.272332 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4"] Jul 19 08:20:56.277889 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.277859 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4w9t4"] Jul 19 08:20:56.731642 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.731608 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" path="/var/lib/kubelet/pods/c680f90b-e292-4c1d-85df-d0029c9bfa80/volumes" Jul 19 08:20:56.731991 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:20:56.731977 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" path="/var/lib/kubelet/pods/e578a50a-a6d6-4887-b62c-f254479d0ba4/volumes" Jul 19 08:21:27.064932 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.064848 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c"] Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065250 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065260 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065268 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065274 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065285 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065291 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065297 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065302 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065367 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065374 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065381 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:27.065582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065388 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.065582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065393 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.065582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.065399 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:27.068515 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.068497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:27.070759 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.070736 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:21:27.076336 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.076310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c"] Jul 19 08:21:27.112014 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.111983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e83affe-e7ec-478e-bc88-c294efb44ff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-psq7c\" (UID: \"0e83affe-e7ec-478e-bc88-c294efb44ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:27.212977 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.212924 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e83affe-e7ec-478e-bc88-c294efb44ff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-psq7c\" (UID: \"0e83affe-e7ec-478e-bc88-c294efb44ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:27.213586 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.213561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e83affe-e7ec-478e-bc88-c294efb44ff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-psq7c\" (UID: \"0e83affe-e7ec-478e-bc88-c294efb44ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:27.381175 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.381136 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:27.512994 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:27.512963 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c"] Jul 19 08:21:27.515785 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:21:27.515757 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e83affe_e7ec_478e_bc88_c294efb44ff3.slice/crio-71872eb604315190359b07bf5c8ce8ecace41d1a3fb0d25b152ec638baab3065 WatchSource:0}: Error finding container 71872eb604315190359b07bf5c8ce8ecace41d1a3fb0d25b152ec638baab3065: Status 404 returned error can't find the container with id 71872eb604315190359b07bf5c8ce8ecace41d1a3fb0d25b152ec638baab3065 Jul 19 08:21:28.403253 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:28.403156 2574 generic.go:356] "Generic (PLEG): container finished" podID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerID="04dce7490dce65cf5b3411cf3114ea7e5ff20737d3279791951ca0be40b243f4" exitCode=6 Jul 19 08:21:28.403612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:28.403248 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" event={"ID":"0e83affe-e7ec-478e-bc88-c294efb44ff3","Type":"ContainerDied","Data":"04dce7490dce65cf5b3411cf3114ea7e5ff20737d3279791951ca0be40b243f4"} Jul 19 08:21:28.403612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:28.403333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" event={"ID":"0e83affe-e7ec-478e-bc88-c294efb44ff3","Type":"ContainerStarted","Data":"71872eb604315190359b07bf5c8ce8ecace41d1a3fb0d25b152ec638baab3065"} Jul 19 08:21:28.403612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:28.403426 2574 scope.go:117] "RemoveContainer" containerID="04dce7490dce65cf5b3411cf3114ea7e5ff20737d3279791951ca0be40b243f4" Jul 19 08:21:29.409867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:29.409832 2574 generic.go:356] "Generic (PLEG): container finished" podID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerID="6a3f2fdfd365eaf091c6be6fb767934030e04179b4b64cf5cff830b5a2a594e3" exitCode=6 Jul 19 08:21:29.410275 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:29.409924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" event={"ID":"0e83affe-e7ec-478e-bc88-c294efb44ff3","Type":"ContainerDied","Data":"6a3f2fdfd365eaf091c6be6fb767934030e04179b4b64cf5cff830b5a2a594e3"} Jul 19 08:21:29.410275 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:29.409990 2574 scope.go:117] "RemoveContainer" containerID="04dce7490dce65cf5b3411cf3114ea7e5ff20737d3279791951ca0be40b243f4" Jul 19 08:21:29.410275 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:29.410221 2574 scope.go:117] "RemoveContainer" containerID="6a3f2fdfd365eaf091c6be6fb767934030e04179b4b64cf5cff830b5a2a594e3" Jul 19 08:21:29.410470 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:29.410452 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-psq7c_odh-ai-gateway-infra(0e83affe-e7ec-478e-bc88-c294efb44ff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" Jul 19 08:21:30.415953 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:30.415920 2574 scope.go:117] "RemoveContainer" containerID="6a3f2fdfd365eaf091c6be6fb767934030e04179b4b64cf5cff830b5a2a594e3" Jul 19 08:21:30.416324 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:30.416147 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-psq7c_odh-ai-gateway-infra(0e83affe-e7ec-478e-bc88-c294efb44ff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" Jul 19 08:21:32.960196 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.960156 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z"] Jul 19 08:21:32.960832 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.960809 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:32.960832 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.960834 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c680f90b-e292-4c1d-85df-d0029c9bfa80" containerName="revoke-keys" Jul 19 08:21:32.960988 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.960857 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:32.960988 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.960867 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e578a50a-a6d6-4887-b62c-f254479d0ba4" containerName="revoke-keys" Jul 19 08:21:32.965566 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.965541 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z"] Jul 19 08:21:32.965750 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:32.965626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:33.067598 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:33.067551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537945e7-6357-47aa-8d8f-261159ccb138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z\" (UID: \"537945e7-6357-47aa-8d8f-261159ccb138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:33.168984 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:33.168892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537945e7-6357-47aa-8d8f-261159ccb138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z\" (UID: \"537945e7-6357-47aa-8d8f-261159ccb138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:33.169529 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:33.169509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537945e7-6357-47aa-8d8f-261159ccb138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z\" (UID: \"537945e7-6357-47aa-8d8f-261159ccb138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:33.276598 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:33.276560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:33.412154 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:33.412130 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z"] Jul 19 08:21:33.414447 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:21:33.414420 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537945e7_6357_47aa_8d8f_261159ccb138.slice/crio-ca945a945690bc0488e41e78c7e2e487e203b8daea4de8229568ced8f9a88995 WatchSource:0}: Error finding container ca945a945690bc0488e41e78c7e2e487e203b8daea4de8229568ced8f9a88995: Status 404 returned error can't find the container with id ca945a945690bc0488e41e78c7e2e487e203b8daea4de8229568ced8f9a88995 Jul 19 08:21:33.430504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:33.430473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" event={"ID":"537945e7-6357-47aa-8d8f-261159ccb138","Type":"ContainerStarted","Data":"ca945a945690bc0488e41e78c7e2e487e203b8daea4de8229568ced8f9a88995"} Jul 19 08:21:34.436537 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:34.436503 2574 generic.go:356] "Generic (PLEG): container finished" podID="537945e7-6357-47aa-8d8f-261159ccb138" containerID="da7108f7115645668c04d789ffa71e6236734663201426776bf345b99a1cc1fc" exitCode=6 Jul 19 08:21:34.436985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:34.436596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" event={"ID":"537945e7-6357-47aa-8d8f-261159ccb138","Type":"ContainerDied","Data":"da7108f7115645668c04d789ffa71e6236734663201426776bf345b99a1cc1fc"} Jul 19 08:21:34.436985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:34.436753 2574 scope.go:117] "RemoveContainer" containerID="da7108f7115645668c04d789ffa71e6236734663201426776bf345b99a1cc1fc" Jul 19 08:21:35.443483 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:35.443446 2574 generic.go:356] "Generic (PLEG): container finished" podID="537945e7-6357-47aa-8d8f-261159ccb138" containerID="954695f7637b63ab5f0ce9ab216426ea68b8ff4424c1672fa8be6f4c16411002" exitCode=6 Jul 19 08:21:35.443930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:35.443528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" event={"ID":"537945e7-6357-47aa-8d8f-261159ccb138","Type":"ContainerDied","Data":"954695f7637b63ab5f0ce9ab216426ea68b8ff4424c1672fa8be6f4c16411002"} Jul 19 08:21:35.443930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:35.443587 2574 scope.go:117] "RemoveContainer" containerID="da7108f7115645668c04d789ffa71e6236734663201426776bf345b99a1cc1fc" Jul 19 08:21:35.443930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:35.443821 2574 scope.go:117] "RemoveContainer" containerID="954695f7637b63ab5f0ce9ab216426ea68b8ff4424c1672fa8be6f4c16411002" Jul 19 08:21:35.444087 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:35.444050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z_odh-ai-gateway-infra(537945e7-6357-47aa-8d8f-261159ccb138)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" podUID="537945e7-6357-47aa-8d8f-261159ccb138" Jul 19 08:21:36.450392 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:36.450360 2574 scope.go:117] "RemoveContainer" containerID="954695f7637b63ab5f0ce9ab216426ea68b8ff4424c1672fa8be6f4c16411002" Jul 19 08:21:36.450917 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:36.450608 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z_odh-ai-gateway-infra(537945e7-6357-47aa-8d8f-261159ccb138)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" podUID="537945e7-6357-47aa-8d8f-261159ccb138" Jul 19 08:21:44.789357 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:44.789316 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5"] Jul 19 08:21:44.794699 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:44.794657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:21:44.798334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:44.798307 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5"] Jul 19 08:21:44.878505 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:44.878460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5\" (UID: \"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:21:44.979364 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:44.979323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5\" (UID: \"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:21:44.980015 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:44.979996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5\" (UID: \"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:21:45.107173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:45.107127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:21:45.238811 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:45.238784 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5"] Jul 19 08:21:45.242308 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:21:45.242272 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6ce47e_c545_40b8_a909_4d2cbcefa1ea.slice/crio-cff424b296cb241fa3fc59ad6e789339044ce61851ea3ba44eeb6b2031c93a16 WatchSource:0}: Error finding container cff424b296cb241fa3fc59ad6e789339044ce61851ea3ba44eeb6b2031c93a16: Status 404 returned error can't find the container with id cff424b296cb241fa3fc59ad6e789339044ce61851ea3ba44eeb6b2031c93a16 Jul 19 08:21:45.493200 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:45.493166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" event={"ID":"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea","Type":"ContainerStarted","Data":"cff424b296cb241fa3fc59ad6e789339044ce61851ea3ba44eeb6b2031c93a16"} Jul 19 08:21:45.726614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:45.726582 2574 scope.go:117] "RemoveContainer" containerID="6a3f2fdfd365eaf091c6be6fb767934030e04179b4b64cf5cff830b5a2a594e3" Jul 19 08:21:46.499927 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.499889 2574 generic.go:356] "Generic (PLEG): container finished" podID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerID="583602e6064e08ac35604d53db528d002d8a30c186011e1cd853de1ff3b5ca19" exitCode=6 Jul 19 08:21:46.500393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.499948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" event={"ID":"0e83affe-e7ec-478e-bc88-c294efb44ff3","Type":"ContainerDied","Data":"583602e6064e08ac35604d53db528d002d8a30c186011e1cd853de1ff3b5ca19"} Jul 19 08:21:46.500393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.500020 2574 scope.go:117] "RemoveContainer" containerID="6a3f2fdfd365eaf091c6be6fb767934030e04179b4b64cf5cff830b5a2a594e3" Jul 19 08:21:46.500393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.500359 2574 scope.go:117] "RemoveContainer" containerID="583602e6064e08ac35604d53db528d002d8a30c186011e1cd853de1ff3b5ca19" Jul 19 08:21:46.500619 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:46.500594 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-psq7c_odh-ai-gateway-infra(0e83affe-e7ec-478e-bc88-c294efb44ff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" Jul 19 08:21:46.501800 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.501775 2574 generic.go:356] "Generic (PLEG): container finished" podID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerID="4d772637194285641be84b29a1406858f1415058b09e8744332741cbb7cd950a" exitCode=6 Jul 19 08:21:46.501909 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.501838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" event={"ID":"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea","Type":"ContainerDied","Data":"4d772637194285641be84b29a1406858f1415058b09e8744332741cbb7cd950a"} Jul 19 08:21:46.502023 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:46.502009 2574 scope.go:117] "RemoveContainer" containerID="4d772637194285641be84b29a1406858f1415058b09e8744332741cbb7cd950a" Jul 19 08:21:47.509455 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.509362 2574 generic.go:356] "Generic (PLEG): container finished" podID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerID="ae66bcfcfd99a30ec0996d30621520055e557c74b46b425335534bfece749ec6" exitCode=6 Jul 19 08:21:47.509455 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.509402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" event={"ID":"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea","Type":"ContainerDied","Data":"ae66bcfcfd99a30ec0996d30621520055e557c74b46b425335534bfece749ec6"} Jul 19 08:21:47.509969 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.509464 2574 scope.go:117] "RemoveContainer" containerID="4d772637194285641be84b29a1406858f1415058b09e8744332741cbb7cd950a" Jul 19 08:21:47.509969 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.509669 2574 scope.go:117] "RemoveContainer" containerID="ae66bcfcfd99a30ec0996d30621520055e557c74b46b425335534bfece749ec6" Jul 19 08:21:47.509969 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:47.509959 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5_odh-ai-gateway-infra(8b6ce47e-c545-40b8-a909-4d2cbcefa1ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" Jul 19 08:21:47.521797 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.521775 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c"] Jul 19 08:21:47.660949 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.660924 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:47.726267 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.726233 2574 scope.go:117] "RemoveContainer" containerID="954695f7637b63ab5f0ce9ab216426ea68b8ff4424c1672fa8be6f4c16411002" Jul 19 08:21:47.802750 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.802719 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e83affe-e7ec-478e-bc88-c294efb44ff3-maas-api-service-ca\") pod \"0e83affe-e7ec-478e-bc88-c294efb44ff3\" (UID: \"0e83affe-e7ec-478e-bc88-c294efb44ff3\") " Jul 19 08:21:47.803121 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.803093 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e83affe-e7ec-478e-bc88-c294efb44ff3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e83affe-e7ec-478e-bc88-c294efb44ff3" (UID: "0e83affe-e7ec-478e-bc88-c294efb44ff3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:21:47.803481 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:47.803442 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e83affe-e7ec-478e-bc88-c294efb44ff3-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:21:48.515969 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.515938 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" Jul 19 08:21:48.515969 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.515947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c" event={"ID":"0e83affe-e7ec-478e-bc88-c294efb44ff3","Type":"ContainerDied","Data":"71872eb604315190359b07bf5c8ce8ecace41d1a3fb0d25b152ec638baab3065"} Jul 19 08:21:48.516504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.516003 2574 scope.go:117] "RemoveContainer" containerID="583602e6064e08ac35604d53db528d002d8a30c186011e1cd853de1ff3b5ca19" Jul 19 08:21:48.518195 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.518156 2574 generic.go:356] "Generic (PLEG): container finished" podID="537945e7-6357-47aa-8d8f-261159ccb138" containerID="77e5fd604212e9c601541ab82b1d2033bf2e466da786994a9877360eed1b7e69" exitCode=6 Jul 19 08:21:48.518317 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.518215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" event={"ID":"537945e7-6357-47aa-8d8f-261159ccb138","Type":"ContainerDied","Data":"77e5fd604212e9c601541ab82b1d2033bf2e466da786994a9877360eed1b7e69"} Jul 19 08:21:48.518531 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.518514 2574 scope.go:117] "RemoveContainer" containerID="77e5fd604212e9c601541ab82b1d2033bf2e466da786994a9877360eed1b7e69" Jul 19 08:21:48.518781 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:48.518743 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z_odh-ai-gateway-infra(537945e7-6357-47aa-8d8f-261159ccb138)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" podUID="537945e7-6357-47aa-8d8f-261159ccb138" Jul 19 08:21:48.520233 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.520216 2574 scope.go:117] "RemoveContainer" containerID="ae66bcfcfd99a30ec0996d30621520055e557c74b46b425335534bfece749ec6" Jul 19 08:21:48.520403 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:21:48.520385 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5_odh-ai-gateway-infra(8b6ce47e-c545-40b8-a909-4d2cbcefa1ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" Jul 19 08:21:48.525639 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.525620 2574 scope.go:117] "RemoveContainer" containerID="954695f7637b63ab5f0ce9ab216426ea68b8ff4424c1672fa8be6f4c16411002" Jul 19 08:21:48.541563 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.541537 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c"] Jul 19 08:21:48.549701 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.547102 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-psq7c"] Jul 19 08:21:48.732064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:48.731974 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" path="/var/lib/kubelet/pods/0e83affe-e7ec-478e-bc88-c294efb44ff3/volumes" Jul 19 08:21:49.539738 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:49.539702 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z"] Jul 19 08:21:49.674274 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:49.674247 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:49.823082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:49.822990 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537945e7-6357-47aa-8d8f-261159ccb138-maas-api-service-ca\") pod \"537945e7-6357-47aa-8d8f-261159ccb138\" (UID: \"537945e7-6357-47aa-8d8f-261159ccb138\") " Jul 19 08:21:49.823352 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:49.823331 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537945e7-6357-47aa-8d8f-261159ccb138-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "537945e7-6357-47aa-8d8f-261159ccb138" (UID: "537945e7-6357-47aa-8d8f-261159ccb138"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:21:49.924474 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:49.924434 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/537945e7-6357-47aa-8d8f-261159ccb138-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:21:50.532014 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:50.531973 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" event={"ID":"537945e7-6357-47aa-8d8f-261159ccb138","Type":"ContainerDied","Data":"ca945a945690bc0488e41e78c7e2e487e203b8daea4de8229568ced8f9a88995"} Jul 19 08:21:50.532014 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:50.532006 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z" Jul 19 08:21:50.532248 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:50.532050 2574 scope.go:117] "RemoveContainer" containerID="77e5fd604212e9c601541ab82b1d2033bf2e466da786994a9877360eed1b7e69" Jul 19 08:21:50.556527 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:50.556488 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z"] Jul 19 08:21:50.560509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:50.560485 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-sjx6z"] Jul 19 08:21:50.732488 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:50.732455 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537945e7-6357-47aa-8d8f-261159ccb138" path="/var/lib/kubelet/pods/537945e7-6357-47aa-8d8f-261159ccb138/volumes" Jul 19 08:21:59.726659 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:21:59.726630 2574 scope.go:117] "RemoveContainer" containerID="ae66bcfcfd99a30ec0996d30621520055e557c74b46b425335534bfece749ec6" Jul 19 08:22:00.578717 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:00.578604 2574 generic.go:356] "Generic (PLEG): container finished" podID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerID="7195a849a5a168357cf32ca289608f35f0ad140ee8e5a8f1a16425fe90fc08ea" exitCode=6 Jul 19 08:22:00.578717 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:00.578669 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" event={"ID":"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea","Type":"ContainerDied","Data":"7195a849a5a168357cf32ca289608f35f0ad140ee8e5a8f1a16425fe90fc08ea"} Jul 19 08:22:00.578953 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:00.578731 2574 scope.go:117] "RemoveContainer" containerID="ae66bcfcfd99a30ec0996d30621520055e557c74b46b425335534bfece749ec6" Jul 19 08:22:00.579020 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:00.579000 2574 scope.go:117] "RemoveContainer" containerID="7195a849a5a168357cf32ca289608f35f0ad140ee8e5a8f1a16425fe90fc08ea" Jul 19 08:22:00.579285 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:00.579264 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5_odh-ai-gateway-infra(8b6ce47e-c545-40b8-a909-4d2cbcefa1ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" Jul 19 08:22:01.601492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:01.601459 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5"] Jul 19 08:22:01.738323 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:01.738297 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:22:01.834362 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:01.834320 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea-maas-api-service-ca\") pod \"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea\" (UID: \"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea\") " Jul 19 08:22:01.834761 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:01.834734 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" (UID: "8b6ce47e-c545-40b8-a909-4d2cbcefa1ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:22:01.935490 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:01.935393 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:22:02.590840 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:02.590813 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" Jul 19 08:22:02.591021 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:02.590808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5" event={"ID":"8b6ce47e-c545-40b8-a909-4d2cbcefa1ea","Type":"ContainerDied","Data":"cff424b296cb241fa3fc59ad6e789339044ce61851ea3ba44eeb6b2031c93a16"} Jul 19 08:22:02.591021 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:02.590957 2574 scope.go:117] "RemoveContainer" containerID="7195a849a5a168357cf32ca289608f35f0ad140ee8e5a8f1a16425fe90fc08ea" Jul 19 08:22:02.614150 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:02.614119 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5"] Jul 19 08:22:02.619506 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:02.619477 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-7fgg5"] Jul 19 08:22:02.733276 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:02.733243 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" path="/var/lib/kubelet/pods/8b6ce47e-c545-40b8-a909-4d2cbcefa1ea/volumes" Jul 19 08:22:22.997890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.997856 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl"] Jul 19 08:22:22.998252 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998209 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998252 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998220 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998252 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998232 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998252 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998237 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998252 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998245 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998252 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998250 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998257 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998262 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998273 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998278 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998288 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998292 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998303 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998308 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998355 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998362 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998369 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998375 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998382 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998389 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:22.998450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:22.998394 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:23.001398 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.001381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:23.003595 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.003576 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:22:23.008564 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.008537 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl"] Jul 19 08:22:23.133459 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.133421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb55e8e-8bf4-411a-96b7-eb7911217350-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl\" (UID: \"0bb55e8e-8bf4-411a-96b7-eb7911217350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:23.234412 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.234371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb55e8e-8bf4-411a-96b7-eb7911217350-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl\" (UID: \"0bb55e8e-8bf4-411a-96b7-eb7911217350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:23.235072 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.235051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb55e8e-8bf4-411a-96b7-eb7911217350-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl\" (UID: \"0bb55e8e-8bf4-411a-96b7-eb7911217350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:23.313802 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.313731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:23.448955 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.448930 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl"] Jul 19 08:22:23.451076 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:22:23.451046 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb55e8e_8bf4_411a_96b7_eb7911217350.slice/crio-2cefd7cd9255f240986a7e79045f498bdcd7434b6ee46b672a12a1bd8ca5bd20 WatchSource:0}: Error finding container 2cefd7cd9255f240986a7e79045f498bdcd7434b6ee46b672a12a1bd8ca5bd20: Status 404 returned error can't find the container with id 2cefd7cd9255f240986a7e79045f498bdcd7434b6ee46b672a12a1bd8ca5bd20 Jul 19 08:22:23.687818 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:23.687788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" event={"ID":"0bb55e8e-8bf4-411a-96b7-eb7911217350","Type":"ContainerStarted","Data":"2cefd7cd9255f240986a7e79045f498bdcd7434b6ee46b672a12a1bd8ca5bd20"} Jul 19 08:22:24.700818 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:24.700780 2574 generic.go:356] "Generic (PLEG): container finished" podID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerID="50a2562d134d5c63b15fa6babd11504f81d13cc5e1c3920c45b0d21cd5177793" exitCode=6 Jul 19 08:22:24.701206 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:24.700869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" event={"ID":"0bb55e8e-8bf4-411a-96b7-eb7911217350","Type":"ContainerDied","Data":"50a2562d134d5c63b15fa6babd11504f81d13cc5e1c3920c45b0d21cd5177793"} Jul 19 08:22:24.701206 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:24.701044 2574 scope.go:117] "RemoveContainer" containerID="50a2562d134d5c63b15fa6babd11504f81d13cc5e1c3920c45b0d21cd5177793" Jul 19 08:22:25.707779 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:25.707745 2574 generic.go:356] "Generic (PLEG): container finished" podID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerID="e0c92876876f4bd509ca4e86f34f52e7629cff806b34f8b20723fb88184e2455" exitCode=6 Jul 19 08:22:25.708237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:25.707850 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" event={"ID":"0bb55e8e-8bf4-411a-96b7-eb7911217350","Type":"ContainerDied","Data":"e0c92876876f4bd509ca4e86f34f52e7629cff806b34f8b20723fb88184e2455"} Jul 19 08:22:25.708237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:25.707916 2574 scope.go:117] "RemoveContainer" containerID="50a2562d134d5c63b15fa6babd11504f81d13cc5e1c3920c45b0d21cd5177793" Jul 19 08:22:25.708237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:25.708105 2574 scope.go:117] "RemoveContainer" containerID="e0c92876876f4bd509ca4e86f34f52e7629cff806b34f8b20723fb88184e2455" Jul 19 08:22:25.708384 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:25.708342 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl_odh-ai-gateway-infra(0bb55e8e-8bf4-411a-96b7-eb7911217350)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" Jul 19 08:22:26.714519 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:26.714494 2574 scope.go:117] "RemoveContainer" containerID="e0c92876876f4bd509ca4e86f34f52e7629cff806b34f8b20723fb88184e2455" Jul 19 08:22:26.714913 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:26.714739 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl_odh-ai-gateway-infra(0bb55e8e-8bf4-411a-96b7-eb7911217350)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" Jul 19 08:22:27.112216 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112174 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx"] Jul 19 08:22:27.112639 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112621 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:27.112766 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112643 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e83affe-e7ec-478e-bc88-c294efb44ff3" containerName="revoke-keys" Jul 19 08:22:27.112766 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112664 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:27.112766 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112691 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:27.112923 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112805 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6ce47e-c545-40b8-a909-4d2cbcefa1ea" containerName="revoke-keys" Jul 19 08:22:27.112923 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.112819 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="537945e7-6357-47aa-8d8f-261159ccb138" containerName="revoke-keys" Jul 19 08:22:27.117056 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.117034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:27.121691 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.121644 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx"] Jul 19 08:22:27.277084 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.277048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de8709d-9493-42ad-9536-71095e89687c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx\" (UID: \"1de8709d-9493-42ad-9536-71095e89687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:27.378453 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.378355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de8709d-9493-42ad-9536-71095e89687c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx\" (UID: \"1de8709d-9493-42ad-9536-71095e89687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:27.379055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.379033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de8709d-9493-42ad-9536-71095e89687c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx\" (UID: \"1de8709d-9493-42ad-9536-71095e89687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:27.428407 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.428356 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:27.565826 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.565800 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx"] Jul 19 08:22:27.567828 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:22:27.567803 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de8709d_9493_42ad_9536_71095e89687c.slice/crio-021be6ee60db2712e607c310604148f8b782e7d3706565b72c56f9d37d77f3bf WatchSource:0}: Error finding container 021be6ee60db2712e607c310604148f8b782e7d3706565b72c56f9d37d77f3bf: Status 404 returned error can't find the container with id 021be6ee60db2712e607c310604148f8b782e7d3706565b72c56f9d37d77f3bf Jul 19 08:22:27.721720 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:27.721620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" event={"ID":"1de8709d-9493-42ad-9536-71095e89687c","Type":"ContainerStarted","Data":"021be6ee60db2712e607c310604148f8b782e7d3706565b72c56f9d37d77f3bf"} Jul 19 08:22:28.727786 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:28.727736 2574 generic.go:356] "Generic (PLEG): container finished" podID="1de8709d-9493-42ad-9536-71095e89687c" containerID="82868f62a9d547b849a07b781a4e112c4f7b3dd40ff13c96ae650cf7ce046b37" exitCode=6 Jul 19 08:22:28.728202 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:28.728081 2574 scope.go:117] "RemoveContainer" containerID="82868f62a9d547b849a07b781a4e112c4f7b3dd40ff13c96ae650cf7ce046b37" Jul 19 08:22:28.732829 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:28.732804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" event={"ID":"1de8709d-9493-42ad-9536-71095e89687c","Type":"ContainerDied","Data":"82868f62a9d547b849a07b781a4e112c4f7b3dd40ff13c96ae650cf7ce046b37"} Jul 19 08:22:29.734385 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:29.734347 2574 generic.go:356] "Generic (PLEG): container finished" podID="1de8709d-9493-42ad-9536-71095e89687c" containerID="216b294f3c664a3ea27e229d6f1e901afc00daa9d585b6091b2cd99abb2f2a75" exitCode=6 Jul 19 08:22:29.734856 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:29.734437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" event={"ID":"1de8709d-9493-42ad-9536-71095e89687c","Type":"ContainerDied","Data":"216b294f3c664a3ea27e229d6f1e901afc00daa9d585b6091b2cd99abb2f2a75"} Jul 19 08:22:29.734856 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:29.734504 2574 scope.go:117] "RemoveContainer" containerID="82868f62a9d547b849a07b781a4e112c4f7b3dd40ff13c96ae650cf7ce046b37" Jul 19 08:22:29.734856 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:29.734740 2574 scope.go:117] "RemoveContainer" containerID="216b294f3c664a3ea27e229d6f1e901afc00daa9d585b6091b2cd99abb2f2a75" Jul 19 08:22:29.735030 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:29.735010 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx_odh-ai-gateway-infra(1de8709d-9493-42ad-9536-71095e89687c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" podUID="1de8709d-9493-42ad-9536-71095e89687c" Jul 19 08:22:30.741423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:30.741343 2574 scope.go:117] "RemoveContainer" containerID="216b294f3c664a3ea27e229d6f1e901afc00daa9d585b6091b2cd99abb2f2a75" Jul 19 08:22:30.741886 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:30.741542 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx_odh-ai-gateway-infra(1de8709d-9493-42ad-9536-71095e89687c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" podUID="1de8709d-9493-42ad-9536-71095e89687c" Jul 19 08:22:34.830233 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:34.830192 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd"] Jul 19 08:22:34.834518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:34.834493 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:34.840563 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:34.840518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd"] Jul 19 08:22:34.947956 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:34.947909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d166367b-b3ea-437c-b5a7-ca243b882383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd\" (UID: \"d166367b-b3ea-437c-b5a7-ca243b882383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:35.049399 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.049356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d166367b-b3ea-437c-b5a7-ca243b882383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd\" (UID: \"d166367b-b3ea-437c-b5a7-ca243b882383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:35.050060 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.050042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d166367b-b3ea-437c-b5a7-ca243b882383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd\" (UID: \"d166367b-b3ea-437c-b5a7-ca243b882383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:35.146785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.146744 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:35.302896 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.302854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd"] Jul 19 08:22:35.305173 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:22:35.305143 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd166367b_b3ea_437c_b5a7_ca243b882383.slice/crio-8363d926de04d9f9bec2755f402642fadfbf2fae1831ba1515c1c26fea662629 WatchSource:0}: Error finding container 8363d926de04d9f9bec2755f402642fadfbf2fae1831ba1515c1c26fea662629: Status 404 returned error can't find the container with id 8363d926de04d9f9bec2755f402642fadfbf2fae1831ba1515c1c26fea662629 Jul 19 08:22:35.769289 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.769175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" event={"ID":"d166367b-b3ea-437c-b5a7-ca243b882383","Type":"ContainerStarted","Data":"aca457d23f83679fae3950b7261dbac9f34b0ee69d5655cb73433c3ca7826e79"} Jul 19 08:22:35.769289 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.769232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" event={"ID":"d166367b-b3ea-437c-b5a7-ca243b882383","Type":"ContainerStarted","Data":"8363d926de04d9f9bec2755f402642fadfbf2fae1831ba1515c1c26fea662629"} Jul 19 08:22:35.784249 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:35.784192 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" podStartSLOduration=1.784176958 podStartE2EDuration="1.784176958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:22:35.782696641 +0000 UTC m=+2403.562359968" watchObservedRunningTime="2026-07-19 08:22:35.784176958 +0000 UTC m=+2403.563840272" Jul 19 08:22:36.775049 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:36.775015 2574 generic.go:356] "Generic (PLEG): container finished" podID="d166367b-b3ea-437c-b5a7-ca243b882383" containerID="aca457d23f83679fae3950b7261dbac9f34b0ee69d5655cb73433c3ca7826e79" exitCode=6 Jul 19 08:22:36.775455 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:36.775108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" event={"ID":"d166367b-b3ea-437c-b5a7-ca243b882383","Type":"ContainerDied","Data":"aca457d23f83679fae3950b7261dbac9f34b0ee69d5655cb73433c3ca7826e79"} Jul 19 08:22:36.775455 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:36.775311 2574 scope.go:117] "RemoveContainer" containerID="aca457d23f83679fae3950b7261dbac9f34b0ee69d5655cb73433c3ca7826e79" Jul 19 08:22:37.781478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:37.781396 2574 generic.go:356] "Generic (PLEG): container finished" podID="d166367b-b3ea-437c-b5a7-ca243b882383" containerID="0a01eea141a84be46854295cba0ff271562626dcdd9930b0ad13420d23d637ba" exitCode=6 Jul 19 08:22:37.781926 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:37.781491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" event={"ID":"d166367b-b3ea-437c-b5a7-ca243b882383","Type":"ContainerDied","Data":"0a01eea141a84be46854295cba0ff271562626dcdd9930b0ad13420d23d637ba"} Jul 19 08:22:37.781926 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:37.781553 2574 scope.go:117] "RemoveContainer" containerID="aca457d23f83679fae3950b7261dbac9f34b0ee69d5655cb73433c3ca7826e79" Jul 19 08:22:37.781926 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:37.781769 2574 scope.go:117] "RemoveContainer" containerID="0a01eea141a84be46854295cba0ff271562626dcdd9930b0ad13420d23d637ba" Jul 19 08:22:37.782069 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:37.782005 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd_odh-ai-gateway-infra(d166367b-b3ea-437c-b5a7-ca243b882383)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" Jul 19 08:22:38.726433 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:38.726398 2574 scope.go:117] "RemoveContainer" containerID="e0c92876876f4bd509ca4e86f34f52e7629cff806b34f8b20723fb88184e2455" Jul 19 08:22:38.788573 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:38.788549 2574 scope.go:117] "RemoveContainer" containerID="0a01eea141a84be46854295cba0ff271562626dcdd9930b0ad13420d23d637ba" Jul 19 08:22:38.789005 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:38.788759 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd_odh-ai-gateway-infra(d166367b-b3ea-437c-b5a7-ca243b882383)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" Jul 19 08:22:39.794403 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:39.794370 2574 generic.go:356] "Generic (PLEG): container finished" podID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerID="eca080e6d858302e67ceee40c423f9531db6577ea8e9e5029cb4fd5fe48fc503" exitCode=6 Jul 19 08:22:39.794846 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:39.794446 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" event={"ID":"0bb55e8e-8bf4-411a-96b7-eb7911217350","Type":"ContainerDied","Data":"eca080e6d858302e67ceee40c423f9531db6577ea8e9e5029cb4fd5fe48fc503"} Jul 19 08:22:39.794846 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:39.794540 2574 scope.go:117] "RemoveContainer" containerID="e0c92876876f4bd509ca4e86f34f52e7629cff806b34f8b20723fb88184e2455" Jul 19 08:22:39.794969 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:39.794897 2574 scope.go:117] "RemoveContainer" containerID="eca080e6d858302e67ceee40c423f9531db6577ea8e9e5029cb4fd5fe48fc503" Jul 19 08:22:39.795111 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:39.795091 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl_odh-ai-gateway-infra(0bb55e8e-8bf4-411a-96b7-eb7911217350)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" Jul 19 08:22:40.817891 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:40.817853 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl"] Jul 19 08:22:40.960335 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:40.960310 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:40.998597 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:40.998566 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb55e8e-8bf4-411a-96b7-eb7911217350-maas-api-service-ca\") pod \"0bb55e8e-8bf4-411a-96b7-eb7911217350\" (UID: \"0bb55e8e-8bf4-411a-96b7-eb7911217350\") " Jul 19 08:22:40.998932 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:40.998908 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb55e8e-8bf4-411a-96b7-eb7911217350-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bb55e8e-8bf4-411a-96b7-eb7911217350" (UID: "0bb55e8e-8bf4-411a-96b7-eb7911217350"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:22:41.099795 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:41.099764 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb55e8e-8bf4-411a-96b7-eb7911217350-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:22:41.807443 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:41.807417 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" Jul 19 08:22:41.807626 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:41.807420 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl" event={"ID":"0bb55e8e-8bf4-411a-96b7-eb7911217350","Type":"ContainerDied","Data":"2cefd7cd9255f240986a7e79045f498bdcd7434b6ee46b672a12a1bd8ca5bd20"} Jul 19 08:22:41.807626 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:41.807540 2574 scope.go:117] "RemoveContainer" containerID="eca080e6d858302e67ceee40c423f9531db6577ea8e9e5029cb4fd5fe48fc503" Jul 19 08:22:41.830395 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:41.830369 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl"] Jul 19 08:22:41.836765 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:41.836742 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-k8jjl"] Jul 19 08:22:42.728876 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:42.728850 2574 scope.go:117] "RemoveContainer" containerID="216b294f3c664a3ea27e229d6f1e901afc00daa9d585b6091b2cd99abb2f2a75" Jul 19 08:22:42.732478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:42.732448 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" path="/var/lib/kubelet/pods/0bb55e8e-8bf4-411a-96b7-eb7911217350/volumes" Jul 19 08:22:43.820141 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:43.820047 2574 generic.go:356] "Generic (PLEG): container finished" podID="1de8709d-9493-42ad-9536-71095e89687c" containerID="4bbe2d0688b86a7d6ca4d8917b1629de3cbbc6d44e61fbf8750478abd78d5d4f" exitCode=6 Jul 19 08:22:43.820141 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:43.820118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" event={"ID":"1de8709d-9493-42ad-9536-71095e89687c","Type":"ContainerDied","Data":"4bbe2d0688b86a7d6ca4d8917b1629de3cbbc6d44e61fbf8750478abd78d5d4f"} Jul 19 08:22:43.820745 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:43.820208 2574 scope.go:117] "RemoveContainer" containerID="216b294f3c664a3ea27e229d6f1e901afc00daa9d585b6091b2cd99abb2f2a75" Jul 19 08:22:43.820745 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:43.820516 2574 scope.go:117] "RemoveContainer" containerID="4bbe2d0688b86a7d6ca4d8917b1629de3cbbc6d44e61fbf8750478abd78d5d4f" Jul 19 08:22:43.820867 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:43.820800 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx_odh-ai-gateway-infra(1de8709d-9493-42ad-9536-71095e89687c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" podUID="1de8709d-9493-42ad-9536-71095e89687c" Jul 19 08:22:44.843115 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:44.843080 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx"] Jul 19 08:22:44.987188 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:44.987163 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:45.032819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.032782 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de8709d-9493-42ad-9536-71095e89687c-maas-api-service-ca\") pod \"1de8709d-9493-42ad-9536-71095e89687c\" (UID: \"1de8709d-9493-42ad-9536-71095e89687c\") " Jul 19 08:22:45.033150 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.033126 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de8709d-9493-42ad-9536-71095e89687c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1de8709d-9493-42ad-9536-71095e89687c" (UID: "1de8709d-9493-42ad-9536-71095e89687c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:22:45.133405 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.133325 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de8709d-9493-42ad-9536-71095e89687c-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:22:45.831729 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.831691 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" Jul 19 08:22:45.831921 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.831669 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx" event={"ID":"1de8709d-9493-42ad-9536-71095e89687c","Type":"ContainerDied","Data":"021be6ee60db2712e607c310604148f8b782e7d3706565b72c56f9d37d77f3bf"} Jul 19 08:22:45.831921 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.831832 2574 scope.go:117] "RemoveContainer" containerID="4bbe2d0688b86a7d6ca4d8917b1629de3cbbc6d44e61fbf8750478abd78d5d4f" Jul 19 08:22:45.853994 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.853957 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx"] Jul 19 08:22:45.861701 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:45.861648 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rj9jx"] Jul 19 08:22:46.732319 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:46.732287 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de8709d-9493-42ad-9536-71095e89687c" path="/var/lib/kubelet/pods/1de8709d-9493-42ad-9536-71095e89687c/volumes" Jul 19 08:22:52.729183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:52.729154 2574 scope.go:117] "RemoveContainer" containerID="0a01eea141a84be46854295cba0ff271562626dcdd9930b0ad13420d23d637ba" Jul 19 08:22:53.869064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:53.869030 2574 generic.go:356] "Generic (PLEG): container finished" podID="d166367b-b3ea-437c-b5a7-ca243b882383" containerID="c834a30dec391660fb3f050b679f9c30ad4d5bc99bd5e53842b5b58c186be259" exitCode=6 Jul 19 08:22:53.869064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:53.869069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" event={"ID":"d166367b-b3ea-437c-b5a7-ca243b882383","Type":"ContainerDied","Data":"c834a30dec391660fb3f050b679f9c30ad4d5bc99bd5e53842b5b58c186be259"} Jul 19 08:22:53.869563 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:53.869119 2574 scope.go:117] "RemoveContainer" containerID="0a01eea141a84be46854295cba0ff271562626dcdd9930b0ad13420d23d637ba" Jul 19 08:22:53.869563 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:53.869379 2574 scope.go:117] "RemoveContainer" containerID="c834a30dec391660fb3f050b679f9c30ad4d5bc99bd5e53842b5b58c186be259" Jul 19 08:22:53.869698 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:22:53.869597 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd_odh-ai-gateway-infra(d166367b-b3ea-437c-b5a7-ca243b882383)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" Jul 19 08:22:54.890572 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:54.890537 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd"] Jul 19 08:22:55.026886 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.026860 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:55.114437 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.114397 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d166367b-b3ea-437c-b5a7-ca243b882383-maas-api-service-ca\") pod \"d166367b-b3ea-437c-b5a7-ca243b882383\" (UID: \"d166367b-b3ea-437c-b5a7-ca243b882383\") " Jul 19 08:22:55.114819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.114795 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d166367b-b3ea-437c-b5a7-ca243b882383-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d166367b-b3ea-437c-b5a7-ca243b882383" (UID: "d166367b-b3ea-437c-b5a7-ca243b882383"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:22:55.215370 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.215283 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d166367b-b3ea-437c-b5a7-ca243b882383-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:22:55.880788 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.880752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" event={"ID":"d166367b-b3ea-437c-b5a7-ca243b882383","Type":"ContainerDied","Data":"8363d926de04d9f9bec2755f402642fadfbf2fae1831ba1515c1c26fea662629"} Jul 19 08:22:55.880976 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.880793 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd" Jul 19 08:22:55.880976 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.880814 2574 scope.go:117] "RemoveContainer" containerID="c834a30dec391660fb3f050b679f9c30ad4d5bc99bd5e53842b5b58c186be259" Jul 19 08:22:55.901953 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.901927 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd"] Jul 19 08:22:55.907563 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:55.907539 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-h78dd"] Jul 19 08:22:56.732641 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:22:56.732604 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" path="/var/lib/kubelet/pods/d166367b-b3ea-437c-b5a7-ca243b882383/volumes" Jul 19 08:23:09.913734 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.913697 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s"] Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914108 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914120 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914133 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914143 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914153 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914158 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914166 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914173 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914180 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914185 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914191 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914196 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914203 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:09.914221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914209 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:09.914645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914275 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:09.914645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914283 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:09.914645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914289 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:09.914645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914300 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914309 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:09.914645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.914315 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:09.917272 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.917254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:09.919778 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.919750 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:23:09.922835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:09.922808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s"] Jul 19 08:23:10.053619 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.053577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b6f39-19cd-4f2a-acf2-da7239b273b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s\" (UID: \"ad8b6f39-19cd-4f2a-acf2-da7239b273b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:10.154708 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.154653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b6f39-19cd-4f2a-acf2-da7239b273b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s\" (UID: \"ad8b6f39-19cd-4f2a-acf2-da7239b273b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:10.155320 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.155298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b6f39-19cd-4f2a-acf2-da7239b273b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s\" (UID: \"ad8b6f39-19cd-4f2a-acf2-da7239b273b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:10.229367 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.229273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:10.364601 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.364573 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s"] Jul 19 08:23:10.367285 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:23:10.367257 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8b6f39_19cd_4f2a_acf2_da7239b273b9.slice/crio-3060853fcf8ed0ea8ffee6cb52406b75d2d139f70a07b7cbd98694c1ea0e7c41 WatchSource:0}: Error finding container 3060853fcf8ed0ea8ffee6cb52406b75d2d139f70a07b7cbd98694c1ea0e7c41: Status 404 returned error can't find the container with id 3060853fcf8ed0ea8ffee6cb52406b75d2d139f70a07b7cbd98694c1ea0e7c41 Jul 19 08:23:10.951208 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.951168 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerStarted","Data":"f73714fbd9f438ab0518f18b390765ee36b6f65e017afb688b48c564eb1a36c2"} Jul 19 08:23:10.951704 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.951223 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerStarted","Data":"3060853fcf8ed0ea8ffee6cb52406b75d2d139f70a07b7cbd98694c1ea0e7c41"} Jul 19 08:23:10.967793 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:10.967744 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" podStartSLOduration=1.967728661 podStartE2EDuration="1.967728661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:23:10.966244888 +0000 UTC m=+2438.745908203" watchObservedRunningTime="2026-07-19 08:23:10.967728661 +0000 UTC m=+2438.747391974" Jul 19 08:23:11.957610 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:11.957573 2574 generic.go:356] "Generic (PLEG): container finished" podID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerID="f73714fbd9f438ab0518f18b390765ee36b6f65e017afb688b48c564eb1a36c2" exitCode=7 Jul 19 08:23:11.958127 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:11.957658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerDied","Data":"f73714fbd9f438ab0518f18b390765ee36b6f65e017afb688b48c564eb1a36c2"} Jul 19 08:23:11.958127 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:11.957932 2574 scope.go:117] "RemoveContainer" containerID="f73714fbd9f438ab0518f18b390765ee36b6f65e017afb688b48c564eb1a36c2" Jul 19 08:23:12.964291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:12.964193 2574 generic.go:356] "Generic (PLEG): container finished" podID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerID="2388b1a1baf55ab7743d11d345a5f5eb52c19ed7c6bc97d66288ebbe028883e8" exitCode=7 Jul 19 08:23:12.964291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:12.964227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerDied","Data":"2388b1a1baf55ab7743d11d345a5f5eb52c19ed7c6bc97d66288ebbe028883e8"} Jul 19 08:23:12.964291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:12.964290 2574 scope.go:117] "RemoveContainer" containerID="f73714fbd9f438ab0518f18b390765ee36b6f65e017afb688b48c564eb1a36c2" Jul 19 08:23:12.964895 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:12.964543 2574 scope.go:117] "RemoveContainer" containerID="2388b1a1baf55ab7743d11d345a5f5eb52c19ed7c6bc97d66288ebbe028883e8" Jul 19 08:23:12.964895 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:23:12.964856 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s_odh-ai-gateway-infra(ad8b6f39-19cd-4f2a-acf2-da7239b273b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" Jul 19 08:23:13.970381 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:13.970354 2574 scope.go:117] "RemoveContainer" containerID="2388b1a1baf55ab7743d11d345a5f5eb52c19ed7c6bc97d66288ebbe028883e8" Jul 19 08:23:13.970843 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:23:13.970533 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s_odh-ai-gateway-infra(ad8b6f39-19cd-4f2a-acf2-da7239b273b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" Jul 19 08:23:17.160377 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160337 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j"] Jul 19 08:23:17.160794 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160729 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:17.160794 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160740 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:17.160794 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160750 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:17.160794 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160756 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:17.160930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160829 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d166367b-b3ea-437c-b5a7-ca243b882383" containerName="revoke-keys" Jul 19 08:23:17.160930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160839 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb55e8e-8bf4-411a-96b7-eb7911217350" containerName="revoke-keys" Jul 19 08:23:17.160930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.160847 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de8709d-9493-42ad-9536-71095e89687c" containerName="revoke-keys" Jul 19 08:23:17.165090 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.165071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:17.174020 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.173817 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j"] Jul 19 08:23:17.319237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.319194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6874cf6-f1eb-489d-9088-8728ba416ae1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j\" (UID: \"c6874cf6-f1eb-489d-9088-8728ba416ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:17.420548 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.420444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6874cf6-f1eb-489d-9088-8728ba416ae1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j\" (UID: \"c6874cf6-f1eb-489d-9088-8728ba416ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:17.421138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.421117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6874cf6-f1eb-489d-9088-8728ba416ae1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j\" (UID: \"c6874cf6-f1eb-489d-9088-8728ba416ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:17.477434 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.477401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:17.611971 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.611937 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j"] Jul 19 08:23:17.615309 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:23:17.615281 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6874cf6_f1eb_489d_9088_8728ba416ae1.slice/crio-9402fdf0cb32d7a9551fba0dfce8f28e4ba5399b0e34b034f46add311b696781 WatchSource:0}: Error finding container 9402fdf0cb32d7a9551fba0dfce8f28e4ba5399b0e34b034f46add311b696781: Status 404 returned error can't find the container with id 9402fdf0cb32d7a9551fba0dfce8f28e4ba5399b0e34b034f46add311b696781 Jul 19 08:23:17.991099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.991065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerStarted","Data":"8c1eadbec9281e2cc1c8fe07c2b3838f9a5f19dab9b257288440c9a896060223"} Jul 19 08:23:17.991301 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:17.991114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerStarted","Data":"9402fdf0cb32d7a9551fba0dfce8f28e4ba5399b0e34b034f46add311b696781"} Jul 19 08:23:18.005842 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:18.005789 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" podStartSLOduration=1.005775776 podStartE2EDuration="1.005775776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:23:18.003967676 +0000 UTC m=+2445.783630991" watchObservedRunningTime="2026-07-19 08:23:18.005775776 +0000 UTC m=+2445.785439089" Jul 19 08:23:18.997605 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:18.997564 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerID="8c1eadbec9281e2cc1c8fe07c2b3838f9a5f19dab9b257288440c9a896060223" exitCode=6 Jul 19 08:23:18.998047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:18.997653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerDied","Data":"8c1eadbec9281e2cc1c8fe07c2b3838f9a5f19dab9b257288440c9a896060223"} Jul 19 08:23:18.998047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:18.997837 2574 scope.go:117] "RemoveContainer" containerID="8c1eadbec9281e2cc1c8fe07c2b3838f9a5f19dab9b257288440c9a896060223" Jul 19 08:23:20.003695 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:20.003645 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerID="6b6853c92efdc5a7797352abe51c245d451fb235675ad4b7ac4232aaa040c5a4" exitCode=6 Jul 19 08:23:20.004119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:20.003716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerDied","Data":"6b6853c92efdc5a7797352abe51c245d451fb235675ad4b7ac4232aaa040c5a4"} Jul 19 08:23:20.004119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:20.003780 2574 scope.go:117] "RemoveContainer" containerID="8c1eadbec9281e2cc1c8fe07c2b3838f9a5f19dab9b257288440c9a896060223" Jul 19 08:23:20.004119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:20.004004 2574 scope.go:117] "RemoveContainer" containerID="6b6853c92efdc5a7797352abe51c245d451fb235675ad4b7ac4232aaa040c5a4" Jul 19 08:23:20.004245 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:23:20.004224 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j_odh-ai-gateway-infra(c6874cf6-f1eb-489d-9088-8728ba416ae1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" Jul 19 08:23:21.010490 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:21.010459 2574 scope.go:117] "RemoveContainer" containerID="6b6853c92efdc5a7797352abe51c245d451fb235675ad4b7ac4232aaa040c5a4" Jul 19 08:23:21.010885 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:23:21.010642 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j_odh-ai-gateway-infra(c6874cf6-f1eb-489d-9088-8728ba416ae1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" Jul 19 08:23:24.726929 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:24.726896 2574 scope.go:117] "RemoveContainer" containerID="2388b1a1baf55ab7743d11d345a5f5eb52c19ed7c6bc97d66288ebbe028883e8" Jul 19 08:23:25.030380 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:25.030296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerStarted","Data":"245565f9f9f882e75127dc2adbef3a0a69a3706b9619878b164191aaa7b511e2"} Jul 19 08:23:26.036638 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:26.036596 2574 generic.go:356] "Generic (PLEG): container finished" podID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerID="245565f9f9f882e75127dc2adbef3a0a69a3706b9619878b164191aaa7b511e2" exitCode=7 Jul 19 08:23:26.037119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:26.036662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerDied","Data":"245565f9f9f882e75127dc2adbef3a0a69a3706b9619878b164191aaa7b511e2"} Jul 19 08:23:26.037119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:26.036746 2574 scope.go:117] "RemoveContainer" containerID="2388b1a1baf55ab7743d11d345a5f5eb52c19ed7c6bc97d66288ebbe028883e8" Jul 19 08:23:26.037119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:26.037073 2574 scope.go:117] "RemoveContainer" containerID="245565f9f9f882e75127dc2adbef3a0a69a3706b9619878b164191aaa7b511e2" Jul 19 08:23:26.037363 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:23:26.037336 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s_odh-ai-gateway-infra(ad8b6f39-19cd-4f2a-acf2-da7239b273b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" Jul 19 08:23:26.053184 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:26.053152 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s"] Jul 19 08:23:27.172154 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:27.172125 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:27.315595 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:27.315495 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b6f39-19cd-4f2a-acf2-da7239b273b9-maas-api-service-ca\") pod \"ad8b6f39-19cd-4f2a-acf2-da7239b273b9\" (UID: \"ad8b6f39-19cd-4f2a-acf2-da7239b273b9\") " Jul 19 08:23:27.315889 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:27.315868 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8b6f39-19cd-4f2a-acf2-da7239b273b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad8b6f39-19cd-4f2a-acf2-da7239b273b9" (UID: "ad8b6f39-19cd-4f2a-acf2-da7239b273b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:23:27.416932 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:27.416899 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b6f39-19cd-4f2a-acf2-da7239b273b9-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:23:28.049101 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:28.049073 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" Jul 19 08:23:28.049254 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:28.049072 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s" event={"ID":"ad8b6f39-19cd-4f2a-acf2-da7239b273b9","Type":"ContainerDied","Data":"3060853fcf8ed0ea8ffee6cb52406b75d2d139f70a07b7cbd98694c1ea0e7c41"} Jul 19 08:23:28.049254 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:28.049191 2574 scope.go:117] "RemoveContainer" containerID="245565f9f9f882e75127dc2adbef3a0a69a3706b9619878b164191aaa7b511e2" Jul 19 08:23:28.070882 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:28.070856 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s"] Jul 19 08:23:28.077510 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:28.077487 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-68j7s"] Jul 19 08:23:28.732317 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:28.732280 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" path="/var/lib/kubelet/pods/ad8b6f39-19cd-4f2a-acf2-da7239b273b9/volumes" Jul 19 08:23:31.726474 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:31.726438 2574 scope.go:117] "RemoveContainer" containerID="6b6853c92efdc5a7797352abe51c245d451fb235675ad4b7ac4232aaa040c5a4" Jul 19 08:23:32.070230 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:32.070144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerStarted","Data":"0158e91c309ac762fd2883b110ae730f2762039f526fd9ff4df6e0df71a00645"} Jul 19 08:23:33.083600 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:33.083567 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerID="0158e91c309ac762fd2883b110ae730f2762039f526fd9ff4df6e0df71a00645" exitCode=6 Jul 19 08:23:33.084022 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:33.083619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerDied","Data":"0158e91c309ac762fd2883b110ae730f2762039f526fd9ff4df6e0df71a00645"} Jul 19 08:23:33.084022 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:33.083709 2574 scope.go:117] "RemoveContainer" containerID="6b6853c92efdc5a7797352abe51c245d451fb235675ad4b7ac4232aaa040c5a4" Jul 19 08:23:33.084135 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:33.084049 2574 scope.go:117] "RemoveContainer" containerID="0158e91c309ac762fd2883b110ae730f2762039f526fd9ff4df6e0df71a00645" Jul 19 08:23:33.084278 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:23:33.084259 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j_odh-ai-gateway-infra(c6874cf6-f1eb-489d-9088-8728ba416ae1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" Jul 19 08:23:33.096142 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:33.096110 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j"] Jul 19 08:23:34.228492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:34.228462 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:34.277667 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:34.277632 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6874cf6-f1eb-489d-9088-8728ba416ae1-maas-api-service-ca\") pod \"c6874cf6-f1eb-489d-9088-8728ba416ae1\" (UID: \"c6874cf6-f1eb-489d-9088-8728ba416ae1\") " Jul 19 08:23:34.277988 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:34.277966 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6874cf6-f1eb-489d-9088-8728ba416ae1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6874cf6-f1eb-489d-9088-8728ba416ae1" (UID: "c6874cf6-f1eb-489d-9088-8728ba416ae1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:23:34.378825 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:34.378738 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6874cf6-f1eb-489d-9088-8728ba416ae1-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:23:35.095963 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:35.095937 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" Jul 19 08:23:35.096147 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:35.095980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j" event={"ID":"c6874cf6-f1eb-489d-9088-8728ba416ae1","Type":"ContainerDied","Data":"9402fdf0cb32d7a9551fba0dfce8f28e4ba5399b0e34b034f46add311b696781"} Jul 19 08:23:35.096147 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:35.096038 2574 scope.go:117] "RemoveContainer" containerID="0158e91c309ac762fd2883b110ae730f2762039f526fd9ff4df6e0df71a00645" Jul 19 08:23:35.112044 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:35.112018 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j"] Jul 19 08:23:35.117025 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:35.117001 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-bvf2j"] Jul 19 08:23:36.731795 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:23:36.731762 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" path="/var/lib/kubelet/pods/c6874cf6-f1eb-489d-9088-8728ba416ae1/volumes" Jul 19 08:24:07.204443 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204406 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7"] Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204765 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204776 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204784 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204790 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204802 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204809 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204841 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204848 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204898 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204905 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204912 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204918 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:07.205045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.204924 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:07.207978 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.207961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:07.210325 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.210298 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:24:07.214344 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.214275 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7"] Jul 19 08:24:07.243973 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.243937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deee8fb-67e4-4a16-8092-e5db2710d38b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7\" (UID: \"2deee8fb-67e4-4a16-8092-e5db2710d38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:07.344751 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.344713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deee8fb-67e4-4a16-8092-e5db2710d38b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7\" (UID: \"2deee8fb-67e4-4a16-8092-e5db2710d38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:07.345354 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.345333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deee8fb-67e4-4a16-8092-e5db2710d38b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7\" (UID: \"2deee8fb-67e4-4a16-8092-e5db2710d38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:07.520104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.520012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:07.650595 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:07.650571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7"] Jul 19 08:24:07.652751 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:24:07.652721 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deee8fb_67e4_4a16_8092_e5db2710d38b.slice/crio-b860676acc15f159bc6efc9efcfeb4dcd5fd94b08890e6fb818b68a84559752b WatchSource:0}: Error finding container b860676acc15f159bc6efc9efcfeb4dcd5fd94b08890e6fb818b68a84559752b: Status 404 returned error can't find the container with id b860676acc15f159bc6efc9efcfeb4dcd5fd94b08890e6fb818b68a84559752b Jul 19 08:24:08.243281 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:08.243191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" event={"ID":"2deee8fb-67e4-4a16-8092-e5db2710d38b","Type":"ContainerStarted","Data":"0f96b0eeae7e511e8767ea03f0b752a9d7d3473f975e58feeae79340a654a910"} Jul 19 08:24:08.243281 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:08.243247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" event={"ID":"2deee8fb-67e4-4a16-8092-e5db2710d38b","Type":"ContainerStarted","Data":"b860676acc15f159bc6efc9efcfeb4dcd5fd94b08890e6fb818b68a84559752b"} Jul 19 08:24:08.258094 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:08.258025 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" podStartSLOduration=1.258006316 podStartE2EDuration="1.258006316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:24:08.256498149 +0000 UTC m=+2496.036161461" watchObservedRunningTime="2026-07-19 08:24:08.258006316 +0000 UTC m=+2496.037669635" Jul 19 08:24:09.249285 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:09.249247 2574 generic.go:356] "Generic (PLEG): container finished" podID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerID="0f96b0eeae7e511e8767ea03f0b752a9d7d3473f975e58feeae79340a654a910" exitCode=6 Jul 19 08:24:09.249736 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:09.249334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" event={"ID":"2deee8fb-67e4-4a16-8092-e5db2710d38b","Type":"ContainerDied","Data":"0f96b0eeae7e511e8767ea03f0b752a9d7d3473f975e58feeae79340a654a910"} Jul 19 08:24:09.249736 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:09.249534 2574 scope.go:117] "RemoveContainer" containerID="0f96b0eeae7e511e8767ea03f0b752a9d7d3473f975e58feeae79340a654a910" Jul 19 08:24:10.255532 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:10.255447 2574 generic.go:356] "Generic (PLEG): container finished" podID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerID="e30a9b07907200755f8db048a7114afa44f03dbb6bac23c2d8dddcfc08d7b8b8" exitCode=6 Jul 19 08:24:10.255532 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:10.255525 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" event={"ID":"2deee8fb-67e4-4a16-8092-e5db2710d38b","Type":"ContainerDied","Data":"e30a9b07907200755f8db048a7114afa44f03dbb6bac23c2d8dddcfc08d7b8b8"} Jul 19 08:24:10.256005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:10.255586 2574 scope.go:117] "RemoveContainer" containerID="0f96b0eeae7e511e8767ea03f0b752a9d7d3473f975e58feeae79340a654a910" Jul 19 08:24:10.256005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:10.255854 2574 scope.go:117] "RemoveContainer" containerID="e30a9b07907200755f8db048a7114afa44f03dbb6bac23c2d8dddcfc08d7b8b8" Jul 19 08:24:10.256080 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:10.256066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7_odh-ai-gateway-infra(2deee8fb-67e4-4a16-8092-e5db2710d38b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" Jul 19 08:24:11.261388 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:11.261361 2574 scope.go:117] "RemoveContainer" containerID="e30a9b07907200755f8db048a7114afa44f03dbb6bac23c2d8dddcfc08d7b8b8" Jul 19 08:24:11.261805 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:11.261548 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7_odh-ai-gateway-infra(2deee8fb-67e4-4a16-8092-e5db2710d38b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" Jul 19 08:24:13.087416 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.087368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm"] Jul 19 08:24:13.087860 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.087756 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:13.087860 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.087768 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8b6f39-19cd-4f2a-acf2-da7239b273b9" containerName="revoke-keys" Jul 19 08:24:13.087860 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.087782 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:13.087860 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.087787 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:13.087860 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.087851 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6874cf6-f1eb-489d-9088-8728ba416ae1" containerName="revoke-keys" Jul 19 08:24:13.092069 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.092052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:13.096390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.096361 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm"] Jul 19 08:24:13.196849 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.196812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0364c-2a1d-4ead-8119-a286f433634d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm\" (UID: \"c7b0364c-2a1d-4ead-8119-a286f433634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:13.298276 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.298241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0364c-2a1d-4ead-8119-a286f433634d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm\" (UID: \"c7b0364c-2a1d-4ead-8119-a286f433634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:13.298966 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.298939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0364c-2a1d-4ead-8119-a286f433634d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm\" (UID: \"c7b0364c-2a1d-4ead-8119-a286f433634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:13.404594 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.404507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:13.535263 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:13.535163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm"] Jul 19 08:24:13.537725 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:24:13.537670 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b0364c_2a1d_4ead_8119_a286f433634d.slice/crio-00d34dc8d86f84ea2e43934debf69c164e6bcb99dd9bdaf6e90b7f516b30b856 WatchSource:0}: Error finding container 00d34dc8d86f84ea2e43934debf69c164e6bcb99dd9bdaf6e90b7f516b30b856: Status 404 returned error can't find the container with id 00d34dc8d86f84ea2e43934debf69c164e6bcb99dd9bdaf6e90b7f516b30b856 Jul 19 08:24:14.277422 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:14.277387 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerID="e77a817d17405b8a63de9f70c91926536f481730a7882056c2d016f1d42e2e89" exitCode=6 Jul 19 08:24:14.277843 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:14.277434 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" event={"ID":"c7b0364c-2a1d-4ead-8119-a286f433634d","Type":"ContainerDied","Data":"e77a817d17405b8a63de9f70c91926536f481730a7882056c2d016f1d42e2e89"} Jul 19 08:24:14.277843 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:14.277486 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" event={"ID":"c7b0364c-2a1d-4ead-8119-a286f433634d","Type":"ContainerStarted","Data":"00d34dc8d86f84ea2e43934debf69c164e6bcb99dd9bdaf6e90b7f516b30b856"} Jul 19 08:24:14.277843 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:14.277700 2574 scope.go:117] "RemoveContainer" containerID="e77a817d17405b8a63de9f70c91926536f481730a7882056c2d016f1d42e2e89" Jul 19 08:24:15.284784 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:15.284695 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerID="e13a947d4d11cf736d9ea229692c09de4d8f6636a0a989d4d27f350131ac8e8a" exitCode=6 Jul 19 08:24:15.284784 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:15.284768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" event={"ID":"c7b0364c-2a1d-4ead-8119-a286f433634d","Type":"ContainerDied","Data":"e13a947d4d11cf736d9ea229692c09de4d8f6636a0a989d4d27f350131ac8e8a"} Jul 19 08:24:15.285234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:15.284828 2574 scope.go:117] "RemoveContainer" containerID="e77a817d17405b8a63de9f70c91926536f481730a7882056c2d016f1d42e2e89" Jul 19 08:24:15.285234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:15.285036 2574 scope.go:117] "RemoveContainer" containerID="e13a947d4d11cf736d9ea229692c09de4d8f6636a0a989d4d27f350131ac8e8a" Jul 19 08:24:15.285314 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:15.285260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm_odh-ai-gateway-infra(c7b0364c-2a1d-4ead-8119-a286f433634d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" Jul 19 08:24:16.291550 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:16.291515 2574 scope.go:117] "RemoveContainer" containerID="e13a947d4d11cf736d9ea229692c09de4d8f6636a0a989d4d27f350131ac8e8a" Jul 19 08:24:16.291956 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:16.291747 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm_odh-ai-gateway-infra(c7b0364c-2a1d-4ead-8119-a286f433634d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" Jul 19 08:24:24.726727 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:24.726615 2574 scope.go:117] "RemoveContainer" containerID="e30a9b07907200755f8db048a7114afa44f03dbb6bac23c2d8dddcfc08d7b8b8" Jul 19 08:24:25.334119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:25.334080 2574 generic.go:356] "Generic (PLEG): container finished" podID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerID="2b219be6630927524e30f7a8a9e915a7c6c376d6b2462afde4c81424a2f40265" exitCode=6 Jul 19 08:24:25.334294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:25.334152 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" event={"ID":"2deee8fb-67e4-4a16-8092-e5db2710d38b","Type":"ContainerDied","Data":"2b219be6630927524e30f7a8a9e915a7c6c376d6b2462afde4c81424a2f40265"} Jul 19 08:24:25.334294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:25.334219 2574 scope.go:117] "RemoveContainer" containerID="e30a9b07907200755f8db048a7114afa44f03dbb6bac23c2d8dddcfc08d7b8b8" Jul 19 08:24:25.334526 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:25.334510 2574 scope.go:117] "RemoveContainer" containerID="2b219be6630927524e30f7a8a9e915a7c6c376d6b2462afde4c81424a2f40265" Jul 19 08:24:25.334831 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:25.334810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7_odh-ai-gateway-infra(2deee8fb-67e4-4a16-8092-e5db2710d38b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" Jul 19 08:24:26.358325 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:26.358292 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7"] Jul 19 08:24:26.486152 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:26.486125 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:26.620714 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:26.620597 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deee8fb-67e4-4a16-8092-e5db2710d38b-maas-api-service-ca\") pod \"2deee8fb-67e4-4a16-8092-e5db2710d38b\" (UID: \"2deee8fb-67e4-4a16-8092-e5db2710d38b\") " Jul 19 08:24:26.620965 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:26.620941 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deee8fb-67e4-4a16-8092-e5db2710d38b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2deee8fb-67e4-4a16-8092-e5db2710d38b" (UID: "2deee8fb-67e4-4a16-8092-e5db2710d38b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:24:26.721915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:26.721879 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deee8fb-67e4-4a16-8092-e5db2710d38b-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:24:27.345494 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:27.345463 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" Jul 19 08:24:27.345709 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:27.345463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7" event={"ID":"2deee8fb-67e4-4a16-8092-e5db2710d38b","Type":"ContainerDied","Data":"b860676acc15f159bc6efc9efcfeb4dcd5fd94b08890e6fb818b68a84559752b"} Jul 19 08:24:27.345709 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:27.345597 2574 scope.go:117] "RemoveContainer" containerID="2b219be6630927524e30f7a8a9e915a7c6c376d6b2462afde4c81424a2f40265" Jul 19 08:24:27.360359 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:27.360325 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7"] Jul 19 08:24:27.362139 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:27.362114 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-7jnt7"] Jul 19 08:24:28.737390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:28.737354 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" path="/var/lib/kubelet/pods/2deee8fb-67e4-4a16-8092-e5db2710d38b/volumes" Jul 19 08:24:29.726938 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:29.726905 2574 scope.go:117] "RemoveContainer" containerID="e13a947d4d11cf736d9ea229692c09de4d8f6636a0a989d4d27f350131ac8e8a" Jul 19 08:24:30.362551 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:30.362517 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerID="e62931f64412f9c687efa6dc1302cc938c820ee6dade8023ecd208696ba4242a" exitCode=6 Jul 19 08:24:30.362945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:30.362575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" event={"ID":"c7b0364c-2a1d-4ead-8119-a286f433634d","Type":"ContainerDied","Data":"e62931f64412f9c687efa6dc1302cc938c820ee6dade8023ecd208696ba4242a"} Jul 19 08:24:30.362945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:30.362628 2574 scope.go:117] "RemoveContainer" containerID="e13a947d4d11cf736d9ea229692c09de4d8f6636a0a989d4d27f350131ac8e8a" Jul 19 08:24:30.362945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:30.362936 2574 scope.go:117] "RemoveContainer" containerID="e62931f64412f9c687efa6dc1302cc938c820ee6dade8023ecd208696ba4242a" Jul 19 08:24:30.363169 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:30.363148 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm_odh-ai-gateway-infra(c7b0364c-2a1d-4ead-8119-a286f433634d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" Jul 19 08:24:31.387128 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:31.387089 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm"] Jul 19 08:24:31.525531 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:31.525505 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:31.665081 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:31.664999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0364c-2a1d-4ead-8119-a286f433634d-maas-api-service-ca\") pod \"c7b0364c-2a1d-4ead-8119-a286f433634d\" (UID: \"c7b0364c-2a1d-4ead-8119-a286f433634d\") " Jul 19 08:24:31.665371 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:31.665343 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b0364c-2a1d-4ead-8119-a286f433634d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7b0364c-2a1d-4ead-8119-a286f433634d" (UID: "c7b0364c-2a1d-4ead-8119-a286f433634d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:24:31.766589 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:31.766553 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b0364c-2a1d-4ead-8119-a286f433634d-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:24:32.373562 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:32.373523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" event={"ID":"c7b0364c-2a1d-4ead-8119-a286f433634d","Type":"ContainerDied","Data":"00d34dc8d86f84ea2e43934debf69c164e6bcb99dd9bdaf6e90b7f516b30b856"} Jul 19 08:24:32.373562 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:32.373552 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm" Jul 19 08:24:32.373826 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:32.373600 2574 scope.go:117] "RemoveContainer" containerID="e62931f64412f9c687efa6dc1302cc938c820ee6dade8023ecd208696ba4242a" Jul 19 08:24:32.395326 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:32.395301 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm"] Jul 19 08:24:32.401113 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:32.401091 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-5vqdm"] Jul 19 08:24:32.732149 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:32.732076 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" path="/var/lib/kubelet/pods/c7b0364c-2a1d-4ead-8119-a286f433634d/volumes" Jul 19 08:24:50.002044 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002002 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5"] Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002431 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002445 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002457 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002463 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002473 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002479 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002486 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:24:50.002504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002491 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:24:50.002785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002550 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:24:50.002785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002560 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:24:50.002785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002565 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:24:50.002785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.002573 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:24:50.005736 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.005709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:24:50.008216 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.008197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:24:50.011913 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.011871 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5"] Jul 19 08:24:50.040151 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.040113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a39a03-cab1-4e52-8a22-d3e23b674f04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5\" (UID: \"80a39a03-cab1-4e52-8a22-d3e23b674f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:24:50.141568 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.141537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a39a03-cab1-4e52-8a22-d3e23b674f04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5\" (UID: \"80a39a03-cab1-4e52-8a22-d3e23b674f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:24:50.142177 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.142156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a39a03-cab1-4e52-8a22-d3e23b674f04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5\" (UID: \"80a39a03-cab1-4e52-8a22-d3e23b674f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:24:50.318267 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.318169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:24:50.447531 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.447508 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5"] Jul 19 08:24:50.449707 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:24:50.449663 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a39a03_cab1_4e52_8a22_d3e23b674f04.slice/crio-ad987bda7f5ce2f93bd7661510510921fbb82de98a59753b3ea616738c745db5 WatchSource:0}: Error finding container ad987bda7f5ce2f93bd7661510510921fbb82de98a59753b3ea616738c745db5: Status 404 returned error can't find the container with id ad987bda7f5ce2f93bd7661510510921fbb82de98a59753b3ea616738c745db5 Jul 19 08:24:50.458445 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:50.458414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" event={"ID":"80a39a03-cab1-4e52-8a22-d3e23b674f04","Type":"ContainerStarted","Data":"ad987bda7f5ce2f93bd7661510510921fbb82de98a59753b3ea616738c745db5"} Jul 19 08:24:51.464754 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:51.464629 2574 generic.go:356] "Generic (PLEG): container finished" podID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerID="5f789768d9a6437b61e2ec1fc53ee004d40c68946b82fefc53c31b2d2abf3b96" exitCode=6 Jul 19 08:24:51.464754 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:51.464716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" event={"ID":"80a39a03-cab1-4e52-8a22-d3e23b674f04","Type":"ContainerDied","Data":"5f789768d9a6437b61e2ec1fc53ee004d40c68946b82fefc53c31b2d2abf3b96"} Jul 19 08:24:51.465272 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:51.464915 2574 scope.go:117] "RemoveContainer" containerID="5f789768d9a6437b61e2ec1fc53ee004d40c68946b82fefc53c31b2d2abf3b96" Jul 19 08:24:52.471771 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:52.471731 2574 generic.go:356] "Generic (PLEG): container finished" podID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerID="4b361b46f07449e12c910f649e6f73e6fb3c9ad15496d58b3fa87209eb2297be" exitCode=6 Jul 19 08:24:52.472263 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:52.471792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" event={"ID":"80a39a03-cab1-4e52-8a22-d3e23b674f04","Type":"ContainerDied","Data":"4b361b46f07449e12c910f649e6f73e6fb3c9ad15496d58b3fa87209eb2297be"} Jul 19 08:24:52.472263 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:52.471845 2574 scope.go:117] "RemoveContainer" containerID="5f789768d9a6437b61e2ec1fc53ee004d40c68946b82fefc53c31b2d2abf3b96" Jul 19 08:24:52.472263 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:52.472022 2574 scope.go:117] "RemoveContainer" containerID="4b361b46f07449e12c910f649e6f73e6fb3c9ad15496d58b3fa87209eb2297be" Jul 19 08:24:52.472437 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:52.472277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5_odh-ai-gateway-infra(80a39a03-cab1-4e52-8a22-d3e23b674f04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" Jul 19 08:24:53.478118 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:24:53.478087 2574 scope.go:117] "RemoveContainer" containerID="4b361b46f07449e12c910f649e6f73e6fb3c9ad15496d58b3fa87209eb2297be" Jul 19 08:24:53.478536 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:24:53.478289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5_odh-ai-gateway-infra(80a39a03-cab1-4e52-8a22-d3e23b674f04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" Jul 19 08:25:07.250445 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250407 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7"] Jul 19 08:25:07.250848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250802 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:25:07.250848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250813 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:25:07.250848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250829 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:25:07.250848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250834 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:25:07.251001 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250894 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deee8fb-67e4-4a16-8092-e5db2710d38b" containerName="revoke-keys" Jul 19 08:25:07.251001 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.250906 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b0364c-2a1d-4ead-8119-a286f433634d" containerName="revoke-keys" Jul 19 08:25:07.255164 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.255146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:07.259816 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.259700 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7"] Jul 19 08:25:07.401010 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.400974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648be8fa-3665-4df5-92a5-cdd133d4644c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7\" (UID: \"648be8fa-3665-4df5-92a5-cdd133d4644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:07.502312 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.502222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648be8fa-3665-4df5-92a5-cdd133d4644c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7\" (UID: \"648be8fa-3665-4df5-92a5-cdd133d4644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:07.502905 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.502885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648be8fa-3665-4df5-92a5-cdd133d4644c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7\" (UID: \"648be8fa-3665-4df5-92a5-cdd133d4644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:07.567612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.567573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:07.699554 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.699525 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7"] Jul 19 08:25:07.702071 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:25:07.702041 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648be8fa_3665_4df5_92a5_cdd133d4644c.slice/crio-57cdcb0b029c17d3213ed6cd4a310cc28adeb337a3e6511149058758ebdb2f5a WatchSource:0}: Error finding container 57cdcb0b029c17d3213ed6cd4a310cc28adeb337a3e6511149058758ebdb2f5a: Status 404 returned error can't find the container with id 57cdcb0b029c17d3213ed6cd4a310cc28adeb337a3e6511149058758ebdb2f5a Jul 19 08:25:07.726438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:07.726420 2574 scope.go:117] "RemoveContainer" containerID="4b361b46f07449e12c910f649e6f73e6fb3c9ad15496d58b3fa87209eb2297be" Jul 19 08:25:08.560024 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.559986 2574 generic.go:356] "Generic (PLEG): container finished" podID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerID="127718ae565787ce8ad4d800dd240a024db7b6af2ee30c32e8844bbdcb8b230a" exitCode=6 Jul 19 08:25:08.560478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.560064 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" event={"ID":"80a39a03-cab1-4e52-8a22-d3e23b674f04","Type":"ContainerDied","Data":"127718ae565787ce8ad4d800dd240a024db7b6af2ee30c32e8844bbdcb8b230a"} Jul 19 08:25:08.560478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.560123 2574 scope.go:117] "RemoveContainer" containerID="4b361b46f07449e12c910f649e6f73e6fb3c9ad15496d58b3fa87209eb2297be" Jul 19 08:25:08.560478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.560416 2574 scope.go:117] "RemoveContainer" containerID="127718ae565787ce8ad4d800dd240a024db7b6af2ee30c32e8844bbdcb8b230a" Jul 19 08:25:08.560692 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:25:08.560656 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5_odh-ai-gateway-infra(80a39a03-cab1-4e52-8a22-d3e23b674f04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" Jul 19 08:25:08.561791 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.561763 2574 generic.go:356] "Generic (PLEG): container finished" podID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerID="a87ac684fa702d46c945bcaf27b16fb1441ff2e1efcb40e430641aca3d5c7ebc" exitCode=6 Jul 19 08:25:08.561918 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.561847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" event={"ID":"648be8fa-3665-4df5-92a5-cdd133d4644c","Type":"ContainerDied","Data":"a87ac684fa702d46c945bcaf27b16fb1441ff2e1efcb40e430641aca3d5c7ebc"} Jul 19 08:25:08.561918 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.561897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" event={"ID":"648be8fa-3665-4df5-92a5-cdd133d4644c","Type":"ContainerStarted","Data":"57cdcb0b029c17d3213ed6cd4a310cc28adeb337a3e6511149058758ebdb2f5a"} Jul 19 08:25:08.562039 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:08.562003 2574 scope.go:117] "RemoveContainer" containerID="a87ac684fa702d46c945bcaf27b16fb1441ff2e1efcb40e430641aca3d5c7ebc" Jul 19 08:25:09.570178 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.570098 2574 generic.go:356] "Generic (PLEG): container finished" podID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerID="a9995287814a5e51e6e9996283c305d97a337ccbcaa59caf26cd8dfc6bf74c02" exitCode=6 Jul 19 08:25:09.570565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.570177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" event={"ID":"648be8fa-3665-4df5-92a5-cdd133d4644c","Type":"ContainerDied","Data":"a9995287814a5e51e6e9996283c305d97a337ccbcaa59caf26cd8dfc6bf74c02"} Jul 19 08:25:09.570565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.570236 2574 scope.go:117] "RemoveContainer" containerID="a87ac684fa702d46c945bcaf27b16fb1441ff2e1efcb40e430641aca3d5c7ebc" Jul 19 08:25:09.570565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.570488 2574 scope.go:117] "RemoveContainer" containerID="a9995287814a5e51e6e9996283c305d97a337ccbcaa59caf26cd8dfc6bf74c02" Jul 19 08:25:09.570715 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:25:09.570671 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7_odh-ai-gateway-infra(648be8fa-3665-4df5-92a5-cdd133d4644c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" Jul 19 08:25:09.585115 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.585087 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5"] Jul 19 08:25:09.740202 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.740174 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:25:09.825761 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.825632 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a39a03-cab1-4e52-8a22-d3e23b674f04-maas-api-service-ca\") pod \"80a39a03-cab1-4e52-8a22-d3e23b674f04\" (UID: \"80a39a03-cab1-4e52-8a22-d3e23b674f04\") " Jul 19 08:25:09.826064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.826038 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a39a03-cab1-4e52-8a22-d3e23b674f04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80a39a03-cab1-4e52-8a22-d3e23b674f04" (UID: "80a39a03-cab1-4e52-8a22-d3e23b674f04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:25:09.926437 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:09.926391 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a39a03-cab1-4e52-8a22-d3e23b674f04-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:25:10.578251 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.578219 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" Jul 19 08:25:10.578763 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.578252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5" event={"ID":"80a39a03-cab1-4e52-8a22-d3e23b674f04","Type":"ContainerDied","Data":"ad987bda7f5ce2f93bd7661510510921fbb82de98a59753b3ea616738c745db5"} Jul 19 08:25:10.578763 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.578341 2574 scope.go:117] "RemoveContainer" containerID="127718ae565787ce8ad4d800dd240a024db7b6af2ee30c32e8844bbdcb8b230a" Jul 19 08:25:10.580184 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.580163 2574 scope.go:117] "RemoveContainer" containerID="a9995287814a5e51e6e9996283c305d97a337ccbcaa59caf26cd8dfc6bf74c02" Jul 19 08:25:10.580406 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:25:10.580383 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7_odh-ai-gateway-infra(648be8fa-3665-4df5-92a5-cdd133d4644c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" Jul 19 08:25:10.609478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.609449 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5"] Jul 19 08:25:10.610807 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.610785 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-h24p5"] Jul 19 08:25:10.733542 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:10.733504 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" path="/var/lib/kubelet/pods/80a39a03-cab1-4e52-8a22-d3e23b674f04/volumes" Jul 19 08:25:21.726390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:21.726359 2574 scope.go:117] "RemoveContainer" containerID="a9995287814a5e51e6e9996283c305d97a337ccbcaa59caf26cd8dfc6bf74c02" Jul 19 08:25:22.640183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:22.640151 2574 generic.go:356] "Generic (PLEG): container finished" podID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerID="fcad2148e4f9ecbd66893eef0a1142467278274ecd4dcf0b2e9325f622a1c2a7" exitCode=6 Jul 19 08:25:22.640183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:22.640190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" event={"ID":"648be8fa-3665-4df5-92a5-cdd133d4644c","Type":"ContainerDied","Data":"fcad2148e4f9ecbd66893eef0a1142467278274ecd4dcf0b2e9325f622a1c2a7"} Jul 19 08:25:22.640430 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:22.640243 2574 scope.go:117] "RemoveContainer" containerID="a9995287814a5e51e6e9996283c305d97a337ccbcaa59caf26cd8dfc6bf74c02" Jul 19 08:25:22.640579 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:22.640558 2574 scope.go:117] "RemoveContainer" containerID="fcad2148e4f9ecbd66893eef0a1142467278274ecd4dcf0b2e9325f622a1c2a7" Jul 19 08:25:22.640841 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:25:22.640822 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7_odh-ai-gateway-infra(648be8fa-3665-4df5-92a5-cdd133d4644c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" Jul 19 08:25:23.663323 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:23.663288 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7"] Jul 19 08:25:23.800820 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:23.800794 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:23.958753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:23.958629 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648be8fa-3665-4df5-92a5-cdd133d4644c-maas-api-service-ca\") pod \"648be8fa-3665-4df5-92a5-cdd133d4644c\" (UID: \"648be8fa-3665-4df5-92a5-cdd133d4644c\") " Jul 19 08:25:23.959021 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:23.958998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648be8fa-3665-4df5-92a5-cdd133d4644c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "648be8fa-3665-4df5-92a5-cdd133d4644c" (UID: "648be8fa-3665-4df5-92a5-cdd133d4644c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:25:24.065126 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.059787 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648be8fa-3665-4df5-92a5-cdd133d4644c-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:25:24.651649 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.651624 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" Jul 19 08:25:24.651649 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.651641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7" event={"ID":"648be8fa-3665-4df5-92a5-cdd133d4644c","Type":"ContainerDied","Data":"57cdcb0b029c17d3213ed6cd4a310cc28adeb337a3e6511149058758ebdb2f5a"} Jul 19 08:25:24.651905 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.651758 2574 scope.go:117] "RemoveContainer" containerID="fcad2148e4f9ecbd66893eef0a1142467278274ecd4dcf0b2e9325f622a1c2a7" Jul 19 08:25:24.673832 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.673802 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7"] Jul 19 08:25:24.679444 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.679418 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rk8k7"] Jul 19 08:25:24.731525 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:24.731489 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" path="/var/lib/kubelet/pods/648be8fa-3665-4df5-92a5-cdd133d4644c/volumes" Jul 19 08:25:57.292393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292307 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp"] Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292706 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292718 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292729 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292734 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292742 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292747 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292761 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292766 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292774 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292780 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292840 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292847 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292854 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292861 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="648be8fa-3665-4df5-92a5-cdd133d4644c" containerName="revoke-keys" Jul 19 08:25:57.292864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.292867 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:25:57.295812 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.295796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:25:57.297898 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.297872 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:25:57.301616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.301589 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp"] Jul 19 08:25:57.371383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.371342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379c47d2-9a8d-43cb-9970-338ccc61af29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp\" (UID: \"379c47d2-9a8d-43cb-9970-338ccc61af29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:25:57.472151 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.472113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379c47d2-9a8d-43cb-9970-338ccc61af29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp\" (UID: \"379c47d2-9a8d-43cb-9970-338ccc61af29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:25:57.472862 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.472841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379c47d2-9a8d-43cb-9970-338ccc61af29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp\" (UID: \"379c47d2-9a8d-43cb-9970-338ccc61af29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:25:57.607511 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.607459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:25:57.741018 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.740993 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp"] Jul 19 08:25:57.804440 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:57.804409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" event={"ID":"379c47d2-9a8d-43cb-9970-338ccc61af29","Type":"ContainerStarted","Data":"01fbab0830f2a48bb7f688c606b27e7d5706ca53edf04fd3c0949f7d50779082"} Jul 19 08:25:58.810631 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:58.810532 2574 generic.go:356] "Generic (PLEG): container finished" podID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerID="e590e074bb5b96d3cf2032322bf3716660aea14bb83d8b56e6c6ddee60c711e5" exitCode=6 Jul 19 08:25:58.810631 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:58.810575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" event={"ID":"379c47d2-9a8d-43cb-9970-338ccc61af29","Type":"ContainerDied","Data":"e590e074bb5b96d3cf2032322bf3716660aea14bb83d8b56e6c6ddee60c711e5"} Jul 19 08:25:58.811080 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:58.810836 2574 scope.go:117] "RemoveContainer" containerID="e590e074bb5b96d3cf2032322bf3716660aea14bb83d8b56e6c6ddee60c711e5" Jul 19 08:25:59.817174 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:59.817084 2574 generic.go:356] "Generic (PLEG): container finished" podID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerID="ac67024e4eb224b0dcedb7eaaccf25a56de2ef4ef00f63bc1e15179a36ac3a11" exitCode=6 Jul 19 08:25:59.817617 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:59.817168 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" event={"ID":"379c47d2-9a8d-43cb-9970-338ccc61af29","Type":"ContainerDied","Data":"ac67024e4eb224b0dcedb7eaaccf25a56de2ef4ef00f63bc1e15179a36ac3a11"} Jul 19 08:25:59.817617 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:59.817234 2574 scope.go:117] "RemoveContainer" containerID="e590e074bb5b96d3cf2032322bf3716660aea14bb83d8b56e6c6ddee60c711e5" Jul 19 08:25:59.817617 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:25:59.817370 2574 scope.go:117] "RemoveContainer" containerID="ac67024e4eb224b0dcedb7eaaccf25a56de2ef4ef00f63bc1e15179a36ac3a11" Jul 19 08:25:59.817617 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:25:59.817531 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp_odh-ai-gateway-infra(379c47d2-9a8d-43cb-9970-338ccc61af29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" Jul 19 08:26:00.823855 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:00.823829 2574 scope.go:117] "RemoveContainer" containerID="ac67024e4eb224b0dcedb7eaaccf25a56de2ef4ef00f63bc1e15179a36ac3a11" Jul 19 08:26:00.824242 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:00.824020 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp_odh-ai-gateway-infra(379c47d2-9a8d-43cb-9970-338ccc61af29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" Jul 19 08:26:03.236666 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.236631 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn"] Jul 19 08:26:03.237237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.237206 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:26:03.237237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.237231 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:26:03.237393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.237286 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a39a03-cab1-4e52-8a22-d3e23b674f04" containerName="revoke-keys" Jul 19 08:26:03.241609 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.241592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:03.246807 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.246780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn"] Jul 19 08:26:03.428784 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.428740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn\" (UID: \"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:03.530320 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.530226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn\" (UID: \"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:03.530930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.530909 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn\" (UID: \"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:03.555233 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.555203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:03.894059 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:03.894008 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn"] Jul 19 08:26:03.896388 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:26:03.896359 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225c6f41_1d3b_4b7b_9f7c_39c94f8e1562.slice/crio-71994816444c81111716501782e120643c8c12917c88aa78e39e3a22556dcf96 WatchSource:0}: Error finding container 71994816444c81111716501782e120643c8c12917c88aa78e39e3a22556dcf96: Status 404 returned error can't find the container with id 71994816444c81111716501782e120643c8c12917c88aa78e39e3a22556dcf96 Jul 19 08:26:04.842979 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:04.842940 2574 generic.go:356] "Generic (PLEG): container finished" podID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerID="14637aed3058be4d8d90f66ac58153e6c9867f0e5ea06c34f6abed373f94eefd" exitCode=6 Jul 19 08:26:04.843356 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:04.843024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" event={"ID":"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562","Type":"ContainerDied","Data":"14637aed3058be4d8d90f66ac58153e6c9867f0e5ea06c34f6abed373f94eefd"} Jul 19 08:26:04.843356 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:04.843075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" event={"ID":"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562","Type":"ContainerStarted","Data":"71994816444c81111716501782e120643c8c12917c88aa78e39e3a22556dcf96"} Jul 19 08:26:04.843356 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:04.843182 2574 scope.go:117] "RemoveContainer" containerID="14637aed3058be4d8d90f66ac58153e6c9867f0e5ea06c34f6abed373f94eefd" Jul 19 08:26:05.849689 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:05.849643 2574 generic.go:356] "Generic (PLEG): container finished" podID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerID="f4d6902759f36e34fd86e120aed727c9514c5cf268e77e53547c9a96382ed2f3" exitCode=6 Jul 19 08:26:05.850076 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:05.849741 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" event={"ID":"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562","Type":"ContainerDied","Data":"f4d6902759f36e34fd86e120aed727c9514c5cf268e77e53547c9a96382ed2f3"} Jul 19 08:26:05.850076 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:05.849805 2574 scope.go:117] "RemoveContainer" containerID="14637aed3058be4d8d90f66ac58153e6c9867f0e5ea06c34f6abed373f94eefd" Jul 19 08:26:05.850076 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:05.850005 2574 scope.go:117] "RemoveContainer" containerID="f4d6902759f36e34fd86e120aed727c9514c5cf268e77e53547c9a96382ed2f3" Jul 19 08:26:05.850230 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:05.850212 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn_odh-ai-gateway-infra(225c6f41-1d3b-4b7b-9f7c-39c94f8e1562)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" Jul 19 08:26:06.856565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:06.856535 2574 scope.go:117] "RemoveContainer" containerID="f4d6902759f36e34fd86e120aed727c9514c5cf268e77e53547c9a96382ed2f3" Jul 19 08:26:06.857012 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:06.856769 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn_odh-ai-gateway-infra(225c6f41-1d3b-4b7b-9f7c-39c94f8e1562)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" Jul 19 08:26:13.726069 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:13.726035 2574 scope.go:117] "RemoveContainer" containerID="ac67024e4eb224b0dcedb7eaaccf25a56de2ef4ef00f63bc1e15179a36ac3a11" Jul 19 08:26:14.895623 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:14.895590 2574 generic.go:356] "Generic (PLEG): container finished" podID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerID="4396ae7315bdbc8b1c0d698ec19540c92cc9d1fbc1ac8249536af3cb39b03d85" exitCode=6 Jul 19 08:26:14.896086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:14.895670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" event={"ID":"379c47d2-9a8d-43cb-9970-338ccc61af29","Type":"ContainerDied","Data":"4396ae7315bdbc8b1c0d698ec19540c92cc9d1fbc1ac8249536af3cb39b03d85"} Jul 19 08:26:14.896086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:14.895748 2574 scope.go:117] "RemoveContainer" containerID="ac67024e4eb224b0dcedb7eaaccf25a56de2ef4ef00f63bc1e15179a36ac3a11" Jul 19 08:26:14.896086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:14.896073 2574 scope.go:117] "RemoveContainer" containerID="4396ae7315bdbc8b1c0d698ec19540c92cc9d1fbc1ac8249536af3cb39b03d85" Jul 19 08:26:14.896359 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:14.896334 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp_odh-ai-gateway-infra(379c47d2-9a8d-43cb-9970-338ccc61af29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" Jul 19 08:26:15.920109 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:15.920072 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp"] Jul 19 08:26:16.056335 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.056311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:26:16.147854 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.147818 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379c47d2-9a8d-43cb-9970-338ccc61af29-maas-api-service-ca\") pod \"379c47d2-9a8d-43cb-9970-338ccc61af29\" (UID: \"379c47d2-9a8d-43cb-9970-338ccc61af29\") " Jul 19 08:26:16.148195 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.148173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379c47d2-9a8d-43cb-9970-338ccc61af29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "379c47d2-9a8d-43cb-9970-338ccc61af29" (UID: "379c47d2-9a8d-43cb-9970-338ccc61af29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:26:16.248742 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.248649 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379c47d2-9a8d-43cb-9970-338ccc61af29-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:26:16.909444 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.909411 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" Jul 19 08:26:16.909612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.909454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp" event={"ID":"379c47d2-9a8d-43cb-9970-338ccc61af29","Type":"ContainerDied","Data":"01fbab0830f2a48bb7f688c606b27e7d5706ca53edf04fd3c0949f7d50779082"} Jul 19 08:26:16.909612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.909518 2574 scope.go:117] "RemoveContainer" containerID="4396ae7315bdbc8b1c0d698ec19540c92cc9d1fbc1ac8249536af3cb39b03d85" Jul 19 08:26:16.924872 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.924846 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp"] Jul 19 08:26:16.930672 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:16.930645 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l6sfp"] Jul 19 08:26:17.727112 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:17.727081 2574 scope.go:117] "RemoveContainer" containerID="f4d6902759f36e34fd86e120aed727c9514c5cf268e77e53547c9a96382ed2f3" Jul 19 08:26:18.732537 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:18.732454 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" path="/var/lib/kubelet/pods/379c47d2-9a8d-43cb-9970-338ccc61af29/volumes" Jul 19 08:26:18.921609 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:18.921570 2574 generic.go:356] "Generic (PLEG): container finished" podID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerID="31901ecc515c95a97be6629c986d02edfbe9b70429ae4a6aee4bc8fc6aaa37f2" exitCode=6 Jul 19 08:26:18.921822 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:18.921641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" event={"ID":"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562","Type":"ContainerDied","Data":"31901ecc515c95a97be6629c986d02edfbe9b70429ae4a6aee4bc8fc6aaa37f2"} Jul 19 08:26:18.921822 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:18.921728 2574 scope.go:117] "RemoveContainer" containerID="f4d6902759f36e34fd86e120aed727c9514c5cf268e77e53547c9a96382ed2f3" Jul 19 08:26:18.922085 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:18.922061 2574 scope.go:117] "RemoveContainer" containerID="31901ecc515c95a97be6629c986d02edfbe9b70429ae4a6aee4bc8fc6aaa37f2" Jul 19 08:26:18.922446 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:18.922422 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn_odh-ai-gateway-infra(225c6f41-1d3b-4b7b-9f7c-39c94f8e1562)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" Jul 19 08:26:19.945313 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:19.945272 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn"] Jul 19 08:26:20.111336 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.111311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:20.184189 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.184159 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562-maas-api-service-ca\") pod \"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562\" (UID: \"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562\") " Jul 19 08:26:20.184532 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.184507 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" (UID: "225c6f41-1d3b-4b7b-9f7c-39c94f8e1562"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:26:20.285719 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.285592 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:26:20.492940 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.492910 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk"] Jul 19 08:26:20.493273 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493262 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493275 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493292 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:20.493322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493297 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:20.493322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493311 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:20.493322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493317 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493331 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493336 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493380 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493386 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493393 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493398 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:20.493469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.493403 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:20.496298 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.496281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:20.502813 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.502783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk"] Jul 19 08:26:20.589086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.589052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8cc50-b19c-4652-83d4-fad0092a19ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk\" (UID: \"f1d8cc50-b19c-4652-83d4-fad0092a19ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:20.690557 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.690521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8cc50-b19c-4652-83d4-fad0092a19ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk\" (UID: \"f1d8cc50-b19c-4652-83d4-fad0092a19ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:20.691197 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.691176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8cc50-b19c-4652-83d4-fad0092a19ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk\" (UID: \"f1d8cc50-b19c-4652-83d4-fad0092a19ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:20.807587 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.807556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:20.936133 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.936093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" event={"ID":"225c6f41-1d3b-4b7b-9f7c-39c94f8e1562","Type":"ContainerDied","Data":"71994816444c81111716501782e120643c8c12917c88aa78e39e3a22556dcf96"} Jul 19 08:26:20.936279 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.936104 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn" Jul 19 08:26:20.936279 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.936194 2574 scope.go:117] "RemoveContainer" containerID="31901ecc515c95a97be6629c986d02edfbe9b70429ae4a6aee4bc8fc6aaa37f2" Jul 19 08:26:20.946504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.946423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk"] Jul 19 08:26:20.948521 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:26:20.948491 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d8cc50_b19c_4652_83d4_fad0092a19ad.slice/crio-05927be87728f27fe3de67485362c00ae1d370058435ef89f497e2adcc37b50f WatchSource:0}: Error finding container 05927be87728f27fe3de67485362c00ae1d370058435ef89f497e2adcc37b50f: Status 404 returned error can't find the container with id 05927be87728f27fe3de67485362c00ae1d370058435ef89f497e2adcc37b50f Jul 19 08:26:20.952391 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.952369 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn"] Jul 19 08:26:20.956950 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:20.956926 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-jxtpn"] Jul 19 08:26:21.944152 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:21.944114 2574 generic.go:356] "Generic (PLEG): container finished" podID="f1d8cc50-b19c-4652-83d4-fad0092a19ad" containerID="8b2ff250f48491847d4600e7dcb5535b13c7cee41cb95a90c4a466abca5a793a" exitCode=0 Jul 19 08:26:21.944317 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:21.944173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" event={"ID":"f1d8cc50-b19c-4652-83d4-fad0092a19ad","Type":"ContainerDied","Data":"8b2ff250f48491847d4600e7dcb5535b13c7cee41cb95a90c4a466abca5a793a"} Jul 19 08:26:21.944317 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:21.944211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" event={"ID":"f1d8cc50-b19c-4652-83d4-fad0092a19ad","Type":"ContainerStarted","Data":"05927be87728f27fe3de67485362c00ae1d370058435ef89f497e2adcc37b50f"} Jul 19 08:26:22.732004 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:22.731971 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" path="/var/lib/kubelet/pods/225c6f41-1d3b-4b7b-9f7c-39c94f8e1562/volumes" Jul 19 08:26:23.085246 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.085219 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:23.213185 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.213151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8cc50-b19c-4652-83d4-fad0092a19ad-maas-api-service-ca\") pod \"f1d8cc50-b19c-4652-83d4-fad0092a19ad\" (UID: \"f1d8cc50-b19c-4652-83d4-fad0092a19ad\") " Jul 19 08:26:23.213541 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.213519 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d8cc50-b19c-4652-83d4-fad0092a19ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1d8cc50-b19c-4652-83d4-fad0092a19ad" (UID: "f1d8cc50-b19c-4652-83d4-fad0092a19ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:26:23.314773 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.314649 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8cc50-b19c-4652-83d4-fad0092a19ad-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:26:23.955369 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.955334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" event={"ID":"f1d8cc50-b19c-4652-83d4-fad0092a19ad","Type":"ContainerDied","Data":"05927be87728f27fe3de67485362c00ae1d370058435ef89f497e2adcc37b50f"} Jul 19 08:26:23.955369 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.955373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk" Jul 19 08:26:23.955838 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:23.955387 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05927be87728f27fe3de67485362c00ae1d370058435ef89f497e2adcc37b50f" Jul 19 08:26:47.331960 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.331919 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95"] Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333235 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333272 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333313 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333322 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c47d2-9a8d-43cb-9970-338ccc61af29" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333353 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d8cc50-b19c-4652-83d4-fad0092a19ad" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333362 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d8cc50-b19c-4652-83d4-fad0092a19ad" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333522 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d8cc50-b19c-4652-83d4-fad0092a19ad" containerName="revoke-keys" Jul 19 08:26:47.335237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.333544 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="225c6f41-1d3b-4b7b-9f7c-39c94f8e1562" containerName="revoke-keys" Jul 19 08:26:47.337358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.337330 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:26:47.340592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.340279 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:26:47.340592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.340312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e7208-b41b-4872-910f-e852c04f4fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-p7h95\" (UID: \"2a1e7208-b41b-4872-910f-e852c04f4fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:26:47.343809 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.343784 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95"] Jul 19 08:26:47.441776 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.441734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e7208-b41b-4872-910f-e852c04f4fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-p7h95\" (UID: \"2a1e7208-b41b-4872-910f-e852c04f4fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:26:47.442370 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.442349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e7208-b41b-4872-910f-e852c04f4fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-p7h95\" (UID: \"2a1e7208-b41b-4872-910f-e852c04f4fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:26:47.651902 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.651845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:26:47.989917 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:47.989891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95"] Jul 19 08:26:47.992223 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:26:47.992195 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1e7208_b41b_4872_910f_e852c04f4fc1.slice/crio-beb9b346d9e7a21eb63cf38ab4e56ed0be76cba7d6938d43e20eeb1a6bccc4bb WatchSource:0}: Error finding container beb9b346d9e7a21eb63cf38ab4e56ed0be76cba7d6938d43e20eeb1a6bccc4bb: Status 404 returned error can't find the container with id beb9b346d9e7a21eb63cf38ab4e56ed0be76cba7d6938d43e20eeb1a6bccc4bb Jul 19 08:26:48.066907 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:48.066871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" event={"ID":"2a1e7208-b41b-4872-910f-e852c04f4fc1","Type":"ContainerStarted","Data":"beb9b346d9e7a21eb63cf38ab4e56ed0be76cba7d6938d43e20eeb1a6bccc4bb"} Jul 19 08:26:49.072741 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:49.072637 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerID="cc446e54b25e1a73d1a85dec28f8e645358c6e253ea4fd01d9f22cfed5da3ae5" exitCode=6 Jul 19 08:26:49.072741 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:49.072710 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" event={"ID":"2a1e7208-b41b-4872-910f-e852c04f4fc1","Type":"ContainerDied","Data":"cc446e54b25e1a73d1a85dec28f8e645358c6e253ea4fd01d9f22cfed5da3ae5"} Jul 19 08:26:49.073183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:49.072953 2574 scope.go:117] "RemoveContainer" containerID="cc446e54b25e1a73d1a85dec28f8e645358c6e253ea4fd01d9f22cfed5da3ae5" Jul 19 08:26:50.079397 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:50.079295 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerID="09972f2b76946cbbf1678ade8b950576798a55fc4f5c6670aad3dbf26f17b79f" exitCode=6 Jul 19 08:26:50.079397 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:50.079376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" event={"ID":"2a1e7208-b41b-4872-910f-e852c04f4fc1","Type":"ContainerDied","Data":"09972f2b76946cbbf1678ade8b950576798a55fc4f5c6670aad3dbf26f17b79f"} Jul 19 08:26:50.079938 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:50.079435 2574 scope.go:117] "RemoveContainer" containerID="cc446e54b25e1a73d1a85dec28f8e645358c6e253ea4fd01d9f22cfed5da3ae5" Jul 19 08:26:50.079938 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:50.079727 2574 scope.go:117] "RemoveContainer" containerID="09972f2b76946cbbf1678ade8b950576798a55fc4f5c6670aad3dbf26f17b79f" Jul 19 08:26:50.080037 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:50.079966 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-p7h95_odh-ai-gateway-infra(2a1e7208-b41b-4872-910f-e852c04f4fc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" Jul 19 08:26:51.085512 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:51.085486 2574 scope.go:117] "RemoveContainer" containerID="09972f2b76946cbbf1678ade8b950576798a55fc4f5c6670aad3dbf26f17b79f" Jul 19 08:26:51.085990 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:51.085777 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-p7h95_odh-ai-gateway-infra(2a1e7208-b41b-4872-910f-e852c04f4fc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" Jul 19 08:26:53.281001 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.280968 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b"] Jul 19 08:26:53.285727 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.285710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:26:53.286984 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.286957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167802c-6467-4acf-959d-55b82a7c81fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b\" (UID: \"7167802c-6467-4acf-959d-55b82a7c81fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:26:53.290650 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.290618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b"] Jul 19 08:26:53.388371 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.388339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167802c-6467-4acf-959d-55b82a7c81fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b\" (UID: \"7167802c-6467-4acf-959d-55b82a7c81fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:26:53.389103 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.389081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167802c-6467-4acf-959d-55b82a7c81fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b\" (UID: \"7167802c-6467-4acf-959d-55b82a7c81fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:26:53.598725 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.598661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:26:53.735296 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:53.735124 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b"] Jul 19 08:26:53.737486 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:26:53.737455 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7167802c_6467_4acf_959d_55b82a7c81fd.slice/crio-e4ed359a98bf691fcdc9ef6c9e2179fdd43f8238a2da8e87e7c9c292f5dce7ea WatchSource:0}: Error finding container e4ed359a98bf691fcdc9ef6c9e2179fdd43f8238a2da8e87e7c9c292f5dce7ea: Status 404 returned error can't find the container with id e4ed359a98bf691fcdc9ef6c9e2179fdd43f8238a2da8e87e7c9c292f5dce7ea Jul 19 08:26:54.099845 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:54.099806 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerStarted","Data":"663b6baf527e7015b506167ca05ed56a6bd212824fe2fda6f3e383264f2108d7"} Jul 19 08:26:54.100027 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:54.099871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerStarted","Data":"e4ed359a98bf691fcdc9ef6c9e2179fdd43f8238a2da8e87e7c9c292f5dce7ea"} Jul 19 08:26:54.116942 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:54.116892 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" podStartSLOduration=1.11687777 podStartE2EDuration="1.11687777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:26:54.11544938 +0000 UTC m=+2661.895112721" watchObservedRunningTime="2026-07-19 08:26:54.11687777 +0000 UTC m=+2661.896541084" Jul 19 08:26:55.106087 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:55.106047 2574 generic.go:356] "Generic (PLEG): container finished" podID="7167802c-6467-4acf-959d-55b82a7c81fd" containerID="663b6baf527e7015b506167ca05ed56a6bd212824fe2fda6f3e383264f2108d7" exitCode=6 Jul 19 08:26:55.106590 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:55.106090 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerDied","Data":"663b6baf527e7015b506167ca05ed56a6bd212824fe2fda6f3e383264f2108d7"} Jul 19 08:26:55.106590 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:55.106349 2574 scope.go:117] "RemoveContainer" containerID="663b6baf527e7015b506167ca05ed56a6bd212824fe2fda6f3e383264f2108d7" Jul 19 08:26:56.112735 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:56.112700 2574 generic.go:356] "Generic (PLEG): container finished" podID="7167802c-6467-4acf-959d-55b82a7c81fd" containerID="5188961a4fdf13fab6a0b915c99f086d3aadc3089c683b00b0b9b5ed30b5c989" exitCode=6 Jul 19 08:26:56.113162 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:56.112782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerDied","Data":"5188961a4fdf13fab6a0b915c99f086d3aadc3089c683b00b0b9b5ed30b5c989"} Jul 19 08:26:56.113162 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:56.112868 2574 scope.go:117] "RemoveContainer" containerID="663b6baf527e7015b506167ca05ed56a6bd212824fe2fda6f3e383264f2108d7" Jul 19 08:26:56.113162 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:56.112939 2574 scope.go:117] "RemoveContainer" containerID="5188961a4fdf13fab6a0b915c99f086d3aadc3089c683b00b0b9b5ed30b5c989" Jul 19 08:26:56.113162 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:56.113130 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b_odh-ai-gateway-infra(7167802c-6467-4acf-959d-55b82a7c81fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" Jul 19 08:26:57.119653 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:26:57.119621 2574 scope.go:117] "RemoveContainer" containerID="5188961a4fdf13fab6a0b915c99f086d3aadc3089c683b00b0b9b5ed30b5c989" Jul 19 08:26:57.120122 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:26:57.119889 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b_odh-ai-gateway-infra(7167802c-6467-4acf-959d-55b82a7c81fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" Jul 19 08:27:04.726627 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:04.726594 2574 scope.go:117] "RemoveContainer" containerID="09972f2b76946cbbf1678ade8b950576798a55fc4f5c6670aad3dbf26f17b79f" Jul 19 08:27:05.044124 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.044039 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp"] Jul 19 08:27:05.047260 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.047243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:05.054359 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.054328 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp"] Jul 19 08:27:05.087152 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.087111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f68c38d-2433-4d5d-b2b9-f35ae73be020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp\" (UID: \"5f68c38d-2433-4d5d-b2b9-f35ae73be020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:05.158749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.158702 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" event={"ID":"2a1e7208-b41b-4872-910f-e852c04f4fc1","Type":"ContainerStarted","Data":"2ee279f8df63f7e96f814a45ad77b65132a8ecaefb1b262a78da09019774b715"} Jul 19 08:27:05.178751 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.178664 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" podStartSLOduration=18.178648017 podStartE2EDuration="18.178648017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:05.178103598 +0000 UTC m=+2672.957766936" watchObservedRunningTime="2026-07-19 08:27:05.178648017 +0000 UTC m=+2672.958311330" Jul 19 08:27:05.187869 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.187838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f68c38d-2433-4d5d-b2b9-f35ae73be020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp\" (UID: \"5f68c38d-2433-4d5d-b2b9-f35ae73be020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:05.188450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.188425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f68c38d-2433-4d5d-b2b9-f35ae73be020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp\" (UID: \"5f68c38d-2433-4d5d-b2b9-f35ae73be020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:05.359217 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.359177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:05.496221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:05.496192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp"] Jul 19 08:27:05.498229 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:27:05.498198 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f68c38d_2433_4d5d_b2b9_f35ae73be020.slice/crio-7581b2964127a3ef6279f7e9a251786711625d2b1a3330a6c7be7445cceef736 WatchSource:0}: Error finding container 7581b2964127a3ef6279f7e9a251786711625d2b1a3330a6c7be7445cceef736: Status 404 returned error can't find the container with id 7581b2964127a3ef6279f7e9a251786711625d2b1a3330a6c7be7445cceef736 Jul 19 08:27:06.165765 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.165725 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerStarted","Data":"af7298bc88641e1f68894a275233f392eae5fe04206e3a31daec2c62430b9af3"} Jul 19 08:27:06.166219 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.165792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerStarted","Data":"7581b2964127a3ef6279f7e9a251786711625d2b1a3330a6c7be7445cceef736"} Jul 19 08:27:06.167565 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.167540 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerID="2ee279f8df63f7e96f814a45ad77b65132a8ecaefb1b262a78da09019774b715" exitCode=6 Jul 19 08:27:06.167706 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.167579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" event={"ID":"2a1e7208-b41b-4872-910f-e852c04f4fc1","Type":"ContainerDied","Data":"2ee279f8df63f7e96f814a45ad77b65132a8ecaefb1b262a78da09019774b715"} Jul 19 08:27:06.167706 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.167620 2574 scope.go:117] "RemoveContainer" containerID="09972f2b76946cbbf1678ade8b950576798a55fc4f5c6670aad3dbf26f17b79f" Jul 19 08:27:06.167889 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.167874 2574 scope.go:117] "RemoveContainer" containerID="2ee279f8df63f7e96f814a45ad77b65132a8ecaefb1b262a78da09019774b715" Jul 19 08:27:06.168124 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:06.168100 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-p7h95_odh-ai-gateway-infra(2a1e7208-b41b-4872-910f-e852c04f4fc1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" Jul 19 08:27:06.180072 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.179883 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" podStartSLOduration=1.179869113 podStartE2EDuration="1.179869113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:06.179357995 +0000 UTC m=+2673.959021311" watchObservedRunningTime="2026-07-19 08:27:06.179869113 +0000 UTC m=+2673.959532426" Jul 19 08:27:06.186996 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:06.186972 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95"] Jul 19 08:27:07.175764 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.175733 2574 generic.go:356] "Generic (PLEG): container finished" podID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerID="af7298bc88641e1f68894a275233f392eae5fe04206e3a31daec2c62430b9af3" exitCode=6 Jul 19 08:27:07.176137 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.175827 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerDied","Data":"af7298bc88641e1f68894a275233f392eae5fe04206e3a31daec2c62430b9af3"} Jul 19 08:27:07.176137 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.176037 2574 scope.go:117] "RemoveContainer" containerID="af7298bc88641e1f68894a275233f392eae5fe04206e3a31daec2c62430b9af3" Jul 19 08:27:07.305089 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.305062 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:27:07.410769 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.410730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e7208-b41b-4872-910f-e852c04f4fc1-maas-api-service-ca\") pod \"2a1e7208-b41b-4872-910f-e852c04f4fc1\" (UID: \"2a1e7208-b41b-4872-910f-e852c04f4fc1\") " Jul 19 08:27:07.411103 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.411077 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1e7208-b41b-4872-910f-e852c04f4fc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a1e7208-b41b-4872-910f-e852c04f4fc1" (UID: "2a1e7208-b41b-4872-910f-e852c04f4fc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:27:07.512123 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:07.512088 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e7208-b41b-4872-910f-e852c04f4fc1-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:27:08.181921 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.181826 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" Jul 19 08:27:08.181921 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.181835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95" event={"ID":"2a1e7208-b41b-4872-910f-e852c04f4fc1","Type":"ContainerDied","Data":"beb9b346d9e7a21eb63cf38ab4e56ed0be76cba7d6938d43e20eeb1a6bccc4bb"} Jul 19 08:27:08.181921 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.181898 2574 scope.go:117] "RemoveContainer" containerID="2ee279f8df63f7e96f814a45ad77b65132a8ecaefb1b262a78da09019774b715" Jul 19 08:27:08.183787 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.183759 2574 generic.go:356] "Generic (PLEG): container finished" podID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerID="5bac868a1644a29480c28f3353002b4af3f04c50b2444ace9480865cbbb2238c" exitCode=6 Jul 19 08:27:08.183915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.183840 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerDied","Data":"5bac868a1644a29480c28f3353002b4af3f04c50b2444ace9480865cbbb2238c"} Jul 19 08:27:08.184028 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.184010 2574 scope.go:117] "RemoveContainer" containerID="5bac868a1644a29480c28f3353002b4af3f04c50b2444ace9480865cbbb2238c" Jul 19 08:27:08.184236 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:08.184217 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp_odh-ai-gateway-infra(5f68c38d-2433-4d5d-b2b9-f35ae73be020)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" Jul 19 08:27:08.190931 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.190917 2574 scope.go:117] "RemoveContainer" containerID="af7298bc88641e1f68894a275233f392eae5fe04206e3a31daec2c62430b9af3" Jul 19 08:27:08.213539 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.213485 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95"] Jul 19 08:27:08.215555 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.215534 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-p7h95"] Jul 19 08:27:08.732393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:08.732358 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" path="/var/lib/kubelet/pods/2a1e7208-b41b-4872-910f-e852c04f4fc1/volumes" Jul 19 08:27:09.190973 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:09.190942 2574 scope.go:117] "RemoveContainer" containerID="5bac868a1644a29480c28f3353002b4af3f04c50b2444ace9480865cbbb2238c" Jul 19 08:27:09.191418 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:09.191156 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp_odh-ai-gateway-infra(5f68c38d-2433-4d5d-b2b9-f35ae73be020)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" Jul 19 08:27:09.726155 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:09.726125 2574 scope.go:117] "RemoveContainer" containerID="5188961a4fdf13fab6a0b915c99f086d3aadc3089c683b00b0b9b5ed30b5c989" Jul 19 08:27:10.198214 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:10.198176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerStarted","Data":"d18172a366b999a85350a46b3407cc264ff74dffd986411ad6d7a8a016bbd049"} Jul 19 08:27:11.204344 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:11.204314 2574 generic.go:356] "Generic (PLEG): container finished" podID="7167802c-6467-4acf-959d-55b82a7c81fd" containerID="d18172a366b999a85350a46b3407cc264ff74dffd986411ad6d7a8a016bbd049" exitCode=6 Jul 19 08:27:11.204781 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:11.204386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerDied","Data":"d18172a366b999a85350a46b3407cc264ff74dffd986411ad6d7a8a016bbd049"} Jul 19 08:27:11.204781 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:11.204448 2574 scope.go:117] "RemoveContainer" containerID="5188961a4fdf13fab6a0b915c99f086d3aadc3089c683b00b0b9b5ed30b5c989" Jul 19 08:27:11.204873 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:11.204834 2574 scope.go:117] "RemoveContainer" containerID="d18172a366b999a85350a46b3407cc264ff74dffd986411ad6d7a8a016bbd049" Jul 19 08:27:11.205057 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:11.205035 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b_odh-ai-gateway-infra(7167802c-6467-4acf-959d-55b82a7c81fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" Jul 19 08:27:11.226270 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:11.226233 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b"] Jul 19 08:27:12.344764 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:12.344737 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:27:12.464085 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:12.463982 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167802c-6467-4acf-959d-55b82a7c81fd-maas-api-service-ca\") pod \"7167802c-6467-4acf-959d-55b82a7c81fd\" (UID: \"7167802c-6467-4acf-959d-55b82a7c81fd\") " Jul 19 08:27:12.464344 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:12.464320 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7167802c-6467-4acf-959d-55b82a7c81fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7167802c-6467-4acf-959d-55b82a7c81fd" (UID: "7167802c-6467-4acf-959d-55b82a7c81fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:27:12.565331 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:12.565286 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167802c-6467-4acf-959d-55b82a7c81fd-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:27:13.215982 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:13.215945 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" event={"ID":"7167802c-6467-4acf-959d-55b82a7c81fd","Type":"ContainerDied","Data":"e4ed359a98bf691fcdc9ef6c9e2179fdd43f8238a2da8e87e7c9c292f5dce7ea"} Jul 19 08:27:13.216174 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:13.216002 2574 scope.go:117] "RemoveContainer" containerID="d18172a366b999a85350a46b3407cc264ff74dffd986411ad6d7a8a016bbd049" Jul 19 08:27:13.216174 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:13.215955 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b" Jul 19 08:27:13.231636 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:13.231612 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b"] Jul 19 08:27:13.238019 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:13.237993 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4cx2b"] Jul 19 08:27:14.731865 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:14.731836 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" path="/var/lib/kubelet/pods/7167802c-6467-4acf-959d-55b82a7c81fd/volumes" Jul 19 08:27:24.727111 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:24.727028 2574 scope.go:117] "RemoveContainer" containerID="5bac868a1644a29480c28f3353002b4af3f04c50b2444ace9480865cbbb2238c" Jul 19 08:27:25.272142 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:25.272041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerStarted","Data":"4ba89f79a6cb3ea32b03c95380ea6587f72470c046f5650aee7cabd2dd02ed86"} Jul 19 08:27:26.279481 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:26.279444 2574 generic.go:356] "Generic (PLEG): container finished" podID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerID="4ba89f79a6cb3ea32b03c95380ea6587f72470c046f5650aee7cabd2dd02ed86" exitCode=6 Jul 19 08:27:26.279481 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:26.279481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerDied","Data":"4ba89f79a6cb3ea32b03c95380ea6587f72470c046f5650aee7cabd2dd02ed86"} Jul 19 08:27:26.280101 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:26.279549 2574 scope.go:117] "RemoveContainer" containerID="5bac868a1644a29480c28f3353002b4af3f04c50b2444ace9480865cbbb2238c" Jul 19 08:27:26.280101 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:26.279955 2574 scope.go:117] "RemoveContainer" containerID="4ba89f79a6cb3ea32b03c95380ea6587f72470c046f5650aee7cabd2dd02ed86" Jul 19 08:27:26.280215 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:26.280162 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp_odh-ai-gateway-infra(5f68c38d-2433-4d5d-b2b9-f35ae73be020)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" Jul 19 08:27:26.295813 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:26.295781 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp"] Jul 19 08:27:27.423164 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:27.423140 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:27.509568 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:27.509534 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f68c38d-2433-4d5d-b2b9-f35ae73be020-maas-api-service-ca\") pod \"5f68c38d-2433-4d5d-b2b9-f35ae73be020\" (UID: \"5f68c38d-2433-4d5d-b2b9-f35ae73be020\") " Jul 19 08:27:27.509861 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:27.509838 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f68c38d-2433-4d5d-b2b9-f35ae73be020-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f68c38d-2433-4d5d-b2b9-f35ae73be020" (UID: "5f68c38d-2433-4d5d-b2b9-f35ae73be020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:27:27.611161 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:27.611122 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f68c38d-2433-4d5d-b2b9-f35ae73be020-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:27:28.292547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:28.292511 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" Jul 19 08:27:28.292723 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:28.292544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp" event={"ID":"5f68c38d-2433-4d5d-b2b9-f35ae73be020","Type":"ContainerDied","Data":"7581b2964127a3ef6279f7e9a251786711625d2b1a3330a6c7be7445cceef736"} Jul 19 08:27:28.292723 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:28.292630 2574 scope.go:117] "RemoveContainer" containerID="4ba89f79a6cb3ea32b03c95380ea6587f72470c046f5650aee7cabd2dd02ed86" Jul 19 08:27:28.319353 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:28.319322 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp"] Jul 19 08:27:28.322631 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:28.322602 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ksvmp"] Jul 19 08:27:28.732396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:28.732364 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" path="/var/lib/kubelet/pods/5f68c38d-2433-4d5d-b2b9-f35ae73be020/volumes" Jul 19 08:27:35.380849 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.380808 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx"] Jul 19 08:27:35.381375 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381356 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.381422 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381381 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.381422 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381399 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381422 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381407 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381422 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381418 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381427 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381444 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381453 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381465 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381473 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381503 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381511 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381522 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:35.381547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381530 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381610 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381621 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381631 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381644 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381653 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381661 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.381852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.381671 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:35.386531 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.386505 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.389475 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.389437 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-07a46e45\"/\"default-dockercfg-jjzdd\"" Jul 19 08:27:35.389789 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.389769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-07a46e45\"/\"e2e-lifecycle-model-07a46e45-kserve-self-signed-certs\"" Jul 19 08:27:35.390228 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.389848 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-07a46e45\"/\"kube-root-ca.crt\"" Jul 19 08:27:35.390228 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.389891 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-07a46e45\"/\"openshift-service-ca.crt\"" Jul 19 08:27:35.391083 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.391064 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx"] Jul 19 08:27:35.482890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.482858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tmp-dir\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.483055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.482899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tls-certs\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.483055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.482941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5g8v\" (UniqueName: \"kubernetes.io/projected/f058ec8e-fe79-4849-86e6-bf77c3f63fca-kube-api-access-h5g8v\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.483055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.482964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-home\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.483175 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.483058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-dshm\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.483175 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.483102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-model-cache\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.583862 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.583825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tmp-dir\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.583875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tls-certs\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.583914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5g8v\" (UniqueName: \"kubernetes.io/projected/f058ec8e-fe79-4849-86e6-bf77c3f63fca-kube-api-access-h5g8v\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.583935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-home\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.584009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-dshm\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.584033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-model-cache\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584321 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.584276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tmp-dir\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584321 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.584310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-model-cache\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.584430 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.584390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-home\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.586349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.586318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-dshm\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.586588 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.586570 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tls-certs\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.591806 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.591781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5g8v\" (UniqueName: \"kubernetes.io/projected/f058ec8e-fe79-4849-86e6-bf77c3f63fca-kube-api-access-h5g8v\") pod \"e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.698464 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.698368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:35.835011 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:35.834987 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx"] Jul 19 08:27:35.837302 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:27:35.837276 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf058ec8e_fe79_4849_86e6_bf77c3f63fca.slice/crio-a927b864ff7e15853eb1d89b8e29ea455fef219925226c260c048b5726d07123 WatchSource:0}: Error finding container a927b864ff7e15853eb1d89b8e29ea455fef219925226c260c048b5726d07123: Status 404 returned error can't find the container with id a927b864ff7e15853eb1d89b8e29ea455fef219925226c260c048b5726d07123 Jul 19 08:27:36.336300 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:36.336265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" event={"ID":"f058ec8e-fe79-4849-86e6-bf77c3f63fca","Type":"ContainerStarted","Data":"ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb"} Jul 19 08:27:36.336484 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:36.336310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" event={"ID":"f058ec8e-fe79-4849-86e6-bf77c3f63fca","Type":"ContainerStarted","Data":"a927b864ff7e15853eb1d89b8e29ea455fef219925226c260c048b5726d07123"} Jul 19 08:27:36.353123 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:36.352990 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" podStartSLOduration=1.3529678330000001 podStartE2EDuration="1.352967833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:36.352603371 +0000 UTC m=+2704.132266685" watchObservedRunningTime="2026-07-19 08:27:36.352967833 +0000 UTC m=+2704.132631150" Jul 19 08:27:43.322913 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.322872 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk"] Jul 19 08:27:43.323305 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.323242 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:43.323305 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.323253 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:43.323305 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.323262 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:43.323305 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.323267 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f68c38d-2433-4d5d-b2b9-f35ae73be020" containerName="revoke-keys" Jul 19 08:27:43.323435 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.323331 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7167802c-6467-4acf-959d-55b82a7c81fd" containerName="revoke-keys" Jul 19 08:27:43.323435 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.323343 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1e7208-b41b-4872-910f-e852c04f4fc1" containerName="revoke-keys" Jul 19 08:27:43.326188 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.326172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:27:43.328456 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.328437 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:27:43.331907 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.331882 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk"] Jul 19 08:27:43.357994 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.357963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb765f12-5ed2-4845-a0c1-136438606229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk\" (UID: \"cb765f12-5ed2-4845-a0c1-136438606229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:27:43.458470 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.458432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb765f12-5ed2-4845-a0c1-136438606229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk\" (UID: \"cb765f12-5ed2-4845-a0c1-136438606229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:27:43.459094 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.459074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb765f12-5ed2-4845-a0c1-136438606229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk\" (UID: \"cb765f12-5ed2-4845-a0c1-136438606229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:27:43.638236 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.638204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:27:43.772452 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:43.772431 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk"] Jul 19 08:27:43.775213 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:27:43.775183 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb765f12_5ed2_4845_a0c1_136438606229.slice/crio-f30a4b689fd3e4cbf98162b6aeb4014f0407ebcf1ba93b4a238510c6bd72954f WatchSource:0}: Error finding container f30a4b689fd3e4cbf98162b6aeb4014f0407ebcf1ba93b4a238510c6bd72954f: Status 404 returned error can't find the container with id f30a4b689fd3e4cbf98162b6aeb4014f0407ebcf1ba93b4a238510c6bd72954f Jul 19 08:27:44.374353 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:44.374315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" event={"ID":"cb765f12-5ed2-4845-a0c1-136438606229","Type":"ContainerStarted","Data":"3e9e7312c275e0b0b889ee67e3fb8f3c00ebb461c43e6b06f5085dbd5618ca6b"} Jul 19 08:27:44.374883 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:44.374373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" event={"ID":"cb765f12-5ed2-4845-a0c1-136438606229","Type":"ContainerStarted","Data":"f30a4b689fd3e4cbf98162b6aeb4014f0407ebcf1ba93b4a238510c6bd72954f"} Jul 19 08:27:44.389229 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:44.389170 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" podStartSLOduration=1.389150393 podStartE2EDuration="1.389150393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:44.387365821 +0000 UTC m=+2712.167029135" watchObservedRunningTime="2026-07-19 08:27:44.389150393 +0000 UTC m=+2712.168813708" Jul 19 08:27:45.380857 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:45.380815 2574 generic.go:356] "Generic (PLEG): container finished" podID="cb765f12-5ed2-4845-a0c1-136438606229" containerID="3e9e7312c275e0b0b889ee67e3fb8f3c00ebb461c43e6b06f5085dbd5618ca6b" exitCode=6 Jul 19 08:27:45.381371 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:45.380901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" event={"ID":"cb765f12-5ed2-4845-a0c1-136438606229","Type":"ContainerDied","Data":"3e9e7312c275e0b0b889ee67e3fb8f3c00ebb461c43e6b06f5085dbd5618ca6b"} Jul 19 08:27:45.381371 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:45.381106 2574 scope.go:117] "RemoveContainer" containerID="3e9e7312c275e0b0b889ee67e3fb8f3c00ebb461c43e6b06f5085dbd5618ca6b" Jul 19 08:27:45.698787 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:45.698671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:45.698787 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:45.698750 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:45.711592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:45.711564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:46.387232 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:46.387200 2574 generic.go:356] "Generic (PLEG): container finished" podID="cb765f12-5ed2-4845-a0c1-136438606229" containerID="bc4a285863eaad0944ae798e6363a703d51a081943aa1c78d957cfb5707ea570" exitCode=6 Jul 19 08:27:46.387749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:46.387233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" event={"ID":"cb765f12-5ed2-4845-a0c1-136438606229","Type":"ContainerDied","Data":"bc4a285863eaad0944ae798e6363a703d51a081943aa1c78d957cfb5707ea570"} Jul 19 08:27:46.387749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:46.387299 2574 scope.go:117] "RemoveContainer" containerID="3e9e7312c275e0b0b889ee67e3fb8f3c00ebb461c43e6b06f5085dbd5618ca6b" Jul 19 08:27:46.387749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:46.387504 2574 scope.go:117] "RemoveContainer" containerID="bc4a285863eaad0944ae798e6363a703d51a081943aa1c78d957cfb5707ea570" Jul 19 08:27:46.387972 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:46.387777 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk_odh-ai-gateway-infra(cb765f12-5ed2-4845-a0c1-136438606229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" podUID="cb765f12-5ed2-4845-a0c1-136438606229" Jul 19 08:27:46.399164 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:46.399146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:27:47.375943 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.375905 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr"] Jul 19 08:27:47.379428 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.379408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:27:47.385707 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.385656 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr"] Jul 19 08:27:47.395617 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.395593 2574 scope.go:117] "RemoveContainer" containerID="bc4a285863eaad0944ae798e6363a703d51a081943aa1c78d957cfb5707ea570" Jul 19 08:27:47.395990 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:47.395815 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk_odh-ai-gateway-infra(cb765f12-5ed2-4845-a0c1-136438606229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" podUID="cb765f12-5ed2-4845-a0c1-136438606229" Jul 19 08:27:47.494406 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.494358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1550238c-0581-42ac-858a-6164ac821389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr\" (UID: \"1550238c-0581-42ac-858a-6164ac821389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:27:47.595844 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.595812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1550238c-0581-42ac-858a-6164ac821389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr\" (UID: \"1550238c-0581-42ac-858a-6164ac821389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:27:47.596448 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.596426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1550238c-0581-42ac-858a-6164ac821389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr\" (UID: \"1550238c-0581-42ac-858a-6164ac821389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:27:47.693915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.693826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:27:47.852690 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:47.852647 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr"] Jul 19 08:27:47.855431 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:27:47.855397 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1550238c_0581_42ac_858a_6164ac821389.slice/crio-892bd378ee7d6573ab73f30900d36be22a79cbc6e803b15db8a2d3cedd4dae88 WatchSource:0}: Error finding container 892bd378ee7d6573ab73f30900d36be22a79cbc6e803b15db8a2d3cedd4dae88: Status 404 returned error can't find the container with id 892bd378ee7d6573ab73f30900d36be22a79cbc6e803b15db8a2d3cedd4dae88 Jul 19 08:27:48.403897 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:48.403808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" event={"ID":"1550238c-0581-42ac-858a-6164ac821389","Type":"ContainerStarted","Data":"fe08f847bcc8584316d8262fbb4423525375d3af2cd9f041ffe5eaa08eb04d24"} Jul 19 08:27:48.403897 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:48.403862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" event={"ID":"1550238c-0581-42ac-858a-6164ac821389","Type":"ContainerStarted","Data":"892bd378ee7d6573ab73f30900d36be22a79cbc6e803b15db8a2d3cedd4dae88"} Jul 19 08:27:48.417726 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:48.417642 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" podStartSLOduration=1.4176252599999999 podStartE2EDuration="1.41762526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:27:48.416534398 +0000 UTC m=+2716.196197731" watchObservedRunningTime="2026-07-19 08:27:48.41762526 +0000 UTC m=+2716.197288574" Jul 19 08:27:49.410372 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:49.410339 2574 generic.go:356] "Generic (PLEG): container finished" podID="1550238c-0581-42ac-858a-6164ac821389" containerID="fe08f847bcc8584316d8262fbb4423525375d3af2cd9f041ffe5eaa08eb04d24" exitCode=6 Jul 19 08:27:49.410798 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:49.410415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" event={"ID":"1550238c-0581-42ac-858a-6164ac821389","Type":"ContainerDied","Data":"fe08f847bcc8584316d8262fbb4423525375d3af2cd9f041ffe5eaa08eb04d24"} Jul 19 08:27:49.410798 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:49.410575 2574 scope.go:117] "RemoveContainer" containerID="fe08f847bcc8584316d8262fbb4423525375d3af2cd9f041ffe5eaa08eb04d24" Jul 19 08:27:50.417237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:50.417199 2574 generic.go:356] "Generic (PLEG): container finished" podID="1550238c-0581-42ac-858a-6164ac821389" containerID="c0033a2245cc98e33647e86858ed47960f7cd6bf8f5d3032776b75d72cd0e8c8" exitCode=6 Jul 19 08:27:50.417644 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:50.417280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" event={"ID":"1550238c-0581-42ac-858a-6164ac821389","Type":"ContainerDied","Data":"c0033a2245cc98e33647e86858ed47960f7cd6bf8f5d3032776b75d72cd0e8c8"} Jul 19 08:27:50.417644 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:50.417337 2574 scope.go:117] "RemoveContainer" containerID="fe08f847bcc8584316d8262fbb4423525375d3af2cd9f041ffe5eaa08eb04d24" Jul 19 08:27:50.417644 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:50.417540 2574 scope.go:117] "RemoveContainer" containerID="c0033a2245cc98e33647e86858ed47960f7cd6bf8f5d3032776b75d72cd0e8c8" Jul 19 08:27:50.417846 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:50.417824 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr_odh-ai-gateway-infra(1550238c-0581-42ac-858a-6164ac821389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" podUID="1550238c-0581-42ac-858a-6164ac821389" Jul 19 08:27:51.423559 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:27:51.423533 2574 scope.go:117] "RemoveContainer" containerID="c0033a2245cc98e33647e86858ed47960f7cd6bf8f5d3032776b75d72cd0e8c8" Jul 19 08:27:51.423954 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:27:51.423748 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr_odh-ai-gateway-infra(1550238c-0581-42ac-858a-6164ac821389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" podUID="1550238c-0581-42ac-858a-6164ac821389" Jul 19 08:28:01.726416 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:01.726379 2574 scope.go:117] "RemoveContainer" containerID="bc4a285863eaad0944ae798e6363a703d51a081943aa1c78d957cfb5707ea570" Jul 19 08:28:02.478688 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:02.478638 2574 generic.go:356] "Generic (PLEG): container finished" podID="cb765f12-5ed2-4845-a0c1-136438606229" containerID="7845d0374929149d107249d6083abe6ef4e3444f7626e42d7e84e3ff829f79f0" exitCode=6 Jul 19 08:28:02.478852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:02.478722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" event={"ID":"cb765f12-5ed2-4845-a0c1-136438606229","Type":"ContainerDied","Data":"7845d0374929149d107249d6083abe6ef4e3444f7626e42d7e84e3ff829f79f0"} Jul 19 08:28:02.478852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:02.478786 2574 scope.go:117] "RemoveContainer" containerID="bc4a285863eaad0944ae798e6363a703d51a081943aa1c78d957cfb5707ea570" Jul 19 08:28:02.479095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:02.479067 2574 scope.go:117] "RemoveContainer" containerID="7845d0374929149d107249d6083abe6ef4e3444f7626e42d7e84e3ff829f79f0" Jul 19 08:28:02.479361 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:02.479333 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk_odh-ai-gateway-infra(cb765f12-5ed2-4845-a0c1-136438606229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" podUID="cb765f12-5ed2-4845-a0c1-136438606229" Jul 19 08:28:03.502595 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:03.502553 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk"] Jul 19 08:28:03.634437 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:03.634412 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:28:03.726971 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:03.726938 2574 scope.go:117] "RemoveContainer" containerID="c0033a2245cc98e33647e86858ed47960f7cd6bf8f5d3032776b75d72cd0e8c8" Jul 19 08:28:03.748316 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:03.748288 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb765f12-5ed2-4845-a0c1-136438606229-maas-api-service-ca\") pod \"cb765f12-5ed2-4845-a0c1-136438606229\" (UID: \"cb765f12-5ed2-4845-a0c1-136438606229\") " Jul 19 08:28:03.748612 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:03.748588 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb765f12-5ed2-4845-a0c1-136438606229-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb765f12-5ed2-4845-a0c1-136438606229" (UID: "cb765f12-5ed2-4845-a0c1-136438606229"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:28:03.850122 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:03.850080 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb765f12-5ed2-4845-a0c1-136438606229-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:04.491173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.491141 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" Jul 19 08:28:04.491173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.491159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk" event={"ID":"cb765f12-5ed2-4845-a0c1-136438606229","Type":"ContainerDied","Data":"f30a4b689fd3e4cbf98162b6aeb4014f0407ebcf1ba93b4a238510c6bd72954f"} Jul 19 08:28:04.491428 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.491215 2574 scope.go:117] "RemoveContainer" containerID="7845d0374929149d107249d6083abe6ef4e3444f7626e42d7e84e3ff829f79f0" Jul 19 08:28:04.493195 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.493171 2574 generic.go:356] "Generic (PLEG): container finished" podID="1550238c-0581-42ac-858a-6164ac821389" containerID="b3817991df09374a7cfd1851a5aa568c4b536ac9bf1e7598970ec3512f41601b" exitCode=6 Jul 19 08:28:04.493345 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.493254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" event={"ID":"1550238c-0581-42ac-858a-6164ac821389","Type":"ContainerDied","Data":"b3817991df09374a7cfd1851a5aa568c4b536ac9bf1e7598970ec3512f41601b"} Jul 19 08:28:04.493577 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.493560 2574 scope.go:117] "RemoveContainer" containerID="b3817991df09374a7cfd1851a5aa568c4b536ac9bf1e7598970ec3512f41601b" Jul 19 08:28:04.493799 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:04.493780 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr_odh-ai-gateway-infra(1550238c-0581-42ac-858a-6164ac821389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" podUID="1550238c-0581-42ac-858a-6164ac821389" Jul 19 08:28:04.500205 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.500186 2574 scope.go:117] "RemoveContainer" containerID="c0033a2245cc98e33647e86858ed47960f7cd6bf8f5d3032776b75d72cd0e8c8" Jul 19 08:28:04.521297 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.521271 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk"] Jul 19 08:28:04.524326 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.524295 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wtspk"] Jul 19 08:28:04.731962 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:04.731930 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb765f12-5ed2-4845-a0c1-136438606229" path="/var/lib/kubelet/pods/cb765f12-5ed2-4845-a0c1-136438606229/volumes" Jul 19 08:28:05.517820 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:05.517779 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr"] Jul 19 08:28:05.655924 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:05.655899 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:28:05.768182 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:05.768080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1550238c-0581-42ac-858a-6164ac821389-maas-api-service-ca\") pod \"1550238c-0581-42ac-858a-6164ac821389\" (UID: \"1550238c-0581-42ac-858a-6164ac821389\") " Jul 19 08:28:05.768461 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:05.768439 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1550238c-0581-42ac-858a-6164ac821389-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1550238c-0581-42ac-858a-6164ac821389" (UID: "1550238c-0581-42ac-858a-6164ac821389"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:28:05.869652 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:05.869610 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1550238c-0581-42ac-858a-6164ac821389-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:06.506492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:06.506459 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" Jul 19 08:28:06.506492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:06.506484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr" event={"ID":"1550238c-0581-42ac-858a-6164ac821389","Type":"ContainerDied","Data":"892bd378ee7d6573ab73f30900d36be22a79cbc6e803b15db8a2d3cedd4dae88"} Jul 19 08:28:06.506899 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:06.506551 2574 scope.go:117] "RemoveContainer" containerID="b3817991df09374a7cfd1851a5aa568c4b536ac9bf1e7598970ec3512f41601b" Jul 19 08:28:06.529918 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:06.529887 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr"] Jul 19 08:28:06.535058 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:06.535030 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-gmrhr"] Jul 19 08:28:06.732506 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:06.732471 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1550238c-0581-42ac-858a-6164ac821389" path="/var/lib/kubelet/pods/1550238c-0581-42ac-858a-6164ac821389/volumes" Jul 19 08:28:26.340102 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:26.340062 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx"] Jul 19 08:28:37.418629 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.418582 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n"] Jul 19 08:28:37.419203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419158 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.419203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419178 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.419203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419190 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419198 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419224 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419233 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419242 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419250 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419274 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419282 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419369 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419379 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419389 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419399 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419409 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1550238c-0581-42ac-858a-6164ac821389" containerName="revoke-keys" Jul 19 08:28:37.419421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.419419 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:37.423800 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.423778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:37.426557 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.426531 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:28:37.428390 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.428362 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n"] Jul 19 08:28:37.459849 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.459809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badd07a9-342e-4d37-b2e2-54cb8b7b8d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n\" (UID: \"badd07a9-342e-4d37-b2e2-54cb8b7b8d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:37.561131 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.561074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badd07a9-342e-4d37-b2e2-54cb8b7b8d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n\" (UID: \"badd07a9-342e-4d37-b2e2-54cb8b7b8d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:37.561747 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.561725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badd07a9-342e-4d37-b2e2-54cb8b7b8d82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n\" (UID: \"badd07a9-342e-4d37-b2e2-54cb8b7b8d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:37.735641 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.735544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:37.868831 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:37.868802 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n"] Jul 19 08:28:37.870880 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:28:37.870854 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadd07a9_342e_4d37_b2e2_54cb8b7b8d82.slice/crio-eab5bd5a13615a06c94708f6a48e24bf18d899a855e2499bdf5fb39a6accbeea WatchSource:0}: Error finding container eab5bd5a13615a06c94708f6a48e24bf18d899a855e2499bdf5fb39a6accbeea: Status 404 returned error can't find the container with id eab5bd5a13615a06c94708f6a48e24bf18d899a855e2499bdf5fb39a6accbeea Jul 19 08:28:38.659045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:38.659009 2574 generic.go:356] "Generic (PLEG): container finished" podID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerID="62b39f8ac3901d89ff9cb7a32d1bac88a2a8b0722d454e9f760fc34d9b4f7079" exitCode=6 Jul 19 08:28:38.659480 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:38.659100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" event={"ID":"badd07a9-342e-4d37-b2e2-54cb8b7b8d82","Type":"ContainerDied","Data":"62b39f8ac3901d89ff9cb7a32d1bac88a2a8b0722d454e9f760fc34d9b4f7079"} Jul 19 08:28:38.659480 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:38.659155 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" event={"ID":"badd07a9-342e-4d37-b2e2-54cb8b7b8d82","Type":"ContainerStarted","Data":"eab5bd5a13615a06c94708f6a48e24bf18d899a855e2499bdf5fb39a6accbeea"} Jul 19 08:28:38.659480 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:38.659292 2574 scope.go:117] "RemoveContainer" containerID="62b39f8ac3901d89ff9cb7a32d1bac88a2a8b0722d454e9f760fc34d9b4f7079" Jul 19 08:28:39.665269 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:39.665184 2574 generic.go:356] "Generic (PLEG): container finished" podID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerID="507255c5eec000688ca39fc3729a55fa63e3949272947e782066e0623febc151" exitCode=6 Jul 19 08:28:39.665660 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:39.665266 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" event={"ID":"badd07a9-342e-4d37-b2e2-54cb8b7b8d82","Type":"ContainerDied","Data":"507255c5eec000688ca39fc3729a55fa63e3949272947e782066e0623febc151"} Jul 19 08:28:39.665660 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:39.665317 2574 scope.go:117] "RemoveContainer" containerID="62b39f8ac3901d89ff9cb7a32d1bac88a2a8b0722d454e9f760fc34d9b4f7079" Jul 19 08:28:39.665660 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:39.665533 2574 scope.go:117] "RemoveContainer" containerID="507255c5eec000688ca39fc3729a55fa63e3949272947e782066e0623febc151" Jul 19 08:28:39.665817 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:39.665769 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n_odh-ai-gateway-infra(badd07a9-342e-4d37-b2e2-54cb8b7b8d82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" Jul 19 08:28:40.672124 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:40.672095 2574 scope.go:117] "RemoveContainer" containerID="507255c5eec000688ca39fc3729a55fa63e3949272947e782066e0623febc151" Jul 19 08:28:40.672513 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:40.672289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n_odh-ai-gateway-infra(badd07a9-342e-4d37-b2e2-54cb8b7b8d82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" Jul 19 08:28:41.360966 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.360921 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" podUID="f058ec8e-fe79-4849-86e6-bf77c3f63fca" containerName="main" containerID="cri-o://ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb" gracePeriod=45 Jul 19 08:28:41.605731 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.605705 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:28:41.677634 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.677538 2574 generic.go:356] "Generic (PLEG): container finished" podID="f058ec8e-fe79-4849-86e6-bf77c3f63fca" containerID="ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb" exitCode=0 Jul 19 08:28:41.677634 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.677585 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" Jul 19 08:28:41.678095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.677631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" event={"ID":"f058ec8e-fe79-4849-86e6-bf77c3f63fca","Type":"ContainerDied","Data":"ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb"} Jul 19 08:28:41.678095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.677706 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx" event={"ID":"f058ec8e-fe79-4849-86e6-bf77c3f63fca","Type":"ContainerDied","Data":"a927b864ff7e15853eb1d89b8e29ea455fef219925226c260c048b5726d07123"} Jul 19 08:28:41.678095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.677728 2574 scope.go:117] "RemoveContainer" containerID="ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb" Jul 19 08:28:41.685933 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.685913 2574 scope.go:117] "RemoveContainer" containerID="ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb" Jul 19 08:28:41.686204 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:41.686182 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb\": container with ID starting with ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb not found: ID does not exist" containerID="ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb" Jul 19 08:28:41.686271 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.686213 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb"} err="failed to get container status \"ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb\": rpc error: code = NotFound desc = could not find container \"ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb\": container with ID starting with ec897bbc9e24140993c1fd819a5c9090ac0dc02514ddc2af0ce6ecfff4cc75eb not found: ID does not exist" Jul 19 08:28:41.700578 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-dshm\") pod \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " Jul 19 08:28:41.700666 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700603 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-model-cache\") pod \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " Jul 19 08:28:41.700666 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700628 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tls-certs\") pod \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " Jul 19 08:28:41.700781 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700672 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tmp-dir\") pod \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " Jul 19 08:28:41.700781 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700749 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5g8v\" (UniqueName: \"kubernetes.io/projected/f058ec8e-fe79-4849-86e6-bf77c3f63fca-kube-api-access-h5g8v\") pod \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " Jul 19 08:28:41.700893 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700790 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-home\") pod \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\" (UID: \"f058ec8e-fe79-4849-86e6-bf77c3f63fca\") " Jul 19 08:28:41.700948 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700902 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-model-cache" (OuterVolumeSpecName: "model-cache") pod "f058ec8e-fe79-4849-86e6-bf77c3f63fca" (UID: "f058ec8e-fe79-4849-86e6-bf77c3f63fca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:28:41.701000 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.700963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f058ec8e-fe79-4849-86e6-bf77c3f63fca" (UID: "f058ec8e-fe79-4849-86e6-bf77c3f63fca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:28:41.701121 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.701097 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-home" (OuterVolumeSpecName: "home") pod "f058ec8e-fe79-4849-86e6-bf77c3f63fca" (UID: "f058ec8e-fe79-4849-86e6-bf77c3f63fca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:28:41.701264 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.701166 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:41.701264 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.701182 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:41.701264 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.701210 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:41.703171 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.703151 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f058ec8e-fe79-4849-86e6-bf77c3f63fca-kube-api-access-h5g8v" (OuterVolumeSpecName: "kube-api-access-h5g8v") pod "f058ec8e-fe79-4849-86e6-bf77c3f63fca" (UID: "f058ec8e-fe79-4849-86e6-bf77c3f63fca"). InnerVolumeSpecName "kube-api-access-h5g8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:28:41.703266 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.703252 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-dshm" (OuterVolumeSpecName: "dshm") pod "f058ec8e-fe79-4849-86e6-bf77c3f63fca" (UID: "f058ec8e-fe79-4849-86e6-bf77c3f63fca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:28:41.703307 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.703280 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f058ec8e-fe79-4849-86e6-bf77c3f63fca" (UID: "f058ec8e-fe79-4849-86e6-bf77c3f63fca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:28:41.802235 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.802200 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f058ec8e-fe79-4849-86e6-bf77c3f63fca-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:41.802235 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.802228 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5g8v\" (UniqueName: \"kubernetes.io/projected/f058ec8e-fe79-4849-86e6-bf77c3f63fca-kube-api-access-h5g8v\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:41.802235 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.802240 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f058ec8e-fe79-4849-86e6-bf77c3f63fca-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:41.998463 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:41.998404 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx"] Jul 19 08:28:42.000378 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:42.000356 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-07a46e45/e2e-lifecycle-model-07a46e45-kserve-7dfbc947b7-pgcwx"] Jul 19 08:28:42.734151 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:42.734118 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f058ec8e-fe79-4849-86e6-bf77c3f63fca" path="/var/lib/kubelet/pods/f058ec8e-fe79-4849-86e6-bf77c3f63fca/volumes" Jul 19 08:28:43.368980 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.368946 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9"] Jul 19 08:28:43.369314 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.369301 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:43.369359 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.369316 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb765f12-5ed2-4845-a0c1-136438606229" containerName="revoke-keys" Jul 19 08:28:43.369359 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.369326 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f058ec8e-fe79-4849-86e6-bf77c3f63fca" containerName="main" Jul 19 08:28:43.369359 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.369332 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f058ec8e-fe79-4849-86e6-bf77c3f63fca" containerName="main" Jul 19 08:28:43.369458 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.369393 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f058ec8e-fe79-4849-86e6-bf77c3f63fca" containerName="main" Jul 19 08:28:43.373831 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.373814 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:28:43.379235 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.379208 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9"] Jul 19 08:28:43.519022 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.518981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e34ae28-4fad-45b0-8f1a-2a0c429a0084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9\" (UID: \"1e34ae28-4fad-45b0-8f1a-2a0c429a0084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:28:43.619868 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.619763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e34ae28-4fad-45b0-8f1a-2a0c429a0084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9\" (UID: \"1e34ae28-4fad-45b0-8f1a-2a0c429a0084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:28:43.620345 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.620325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e34ae28-4fad-45b0-8f1a-2a0c429a0084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9\" (UID: \"1e34ae28-4fad-45b0-8f1a-2a0c429a0084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:28:43.685401 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.685375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:28:43.820833 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:43.820713 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9"] Jul 19 08:28:43.823668 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:28:43.823634 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e34ae28_4fad_45b0_8f1a_2a0c429a0084.slice/crio-5becafd96daf9c9e3ecd16851e84e2edf05d1487ffb4689b73b86ead1ead4bef WatchSource:0}: Error finding container 5becafd96daf9c9e3ecd16851e84e2edf05d1487ffb4689b73b86ead1ead4bef: Status 404 returned error can't find the container with id 5becafd96daf9c9e3ecd16851e84e2edf05d1487ffb4689b73b86ead1ead4bef Jul 19 08:28:44.695469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:44.695376 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerID="fee8c7c398d67fcf729f979c148704de95ad2b50a1a2cbae2eb4bf537dfb1ce5" exitCode=6 Jul 19 08:28:44.695625 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:44.695466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" event={"ID":"1e34ae28-4fad-45b0-8f1a-2a0c429a0084","Type":"ContainerDied","Data":"fee8c7c398d67fcf729f979c148704de95ad2b50a1a2cbae2eb4bf537dfb1ce5"} Jul 19 08:28:44.695625 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:44.695525 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" event={"ID":"1e34ae28-4fad-45b0-8f1a-2a0c429a0084","Type":"ContainerStarted","Data":"5becafd96daf9c9e3ecd16851e84e2edf05d1487ffb4689b73b86ead1ead4bef"} Jul 19 08:28:44.695737 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:44.695652 2574 scope.go:117] "RemoveContainer" containerID="fee8c7c398d67fcf729f979c148704de95ad2b50a1a2cbae2eb4bf537dfb1ce5" Jul 19 08:28:45.703028 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:45.702988 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerID="49f3f1b313aaa3b82a184ceadcf6ad167156b19d3eec9d539cd34841c6e14c70" exitCode=6 Jul 19 08:28:45.703434 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:45.703073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" event={"ID":"1e34ae28-4fad-45b0-8f1a-2a0c429a0084","Type":"ContainerDied","Data":"49f3f1b313aaa3b82a184ceadcf6ad167156b19d3eec9d539cd34841c6e14c70"} Jul 19 08:28:45.703434 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:45.703134 2574 scope.go:117] "RemoveContainer" containerID="fee8c7c398d67fcf729f979c148704de95ad2b50a1a2cbae2eb4bf537dfb1ce5" Jul 19 08:28:45.703434 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:45.703389 2574 scope.go:117] "RemoveContainer" containerID="49f3f1b313aaa3b82a184ceadcf6ad167156b19d3eec9d539cd34841c6e14c70" Jul 19 08:28:45.703643 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:45.703623 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9_odh-ai-gateway-infra(1e34ae28-4fad-45b0-8f1a-2a0c429a0084)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" Jul 19 08:28:46.715835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:46.715803 2574 scope.go:117] "RemoveContainer" containerID="49f3f1b313aaa3b82a184ceadcf6ad167156b19d3eec9d539cd34841c6e14c70" Jul 19 08:28:46.716247 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:46.716040 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9_odh-ai-gateway-infra(1e34ae28-4fad-45b0-8f1a-2a0c429a0084)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" Jul 19 08:28:54.726233 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:54.726153 2574 scope.go:117] "RemoveContainer" containerID="507255c5eec000688ca39fc3729a55fa63e3949272947e782066e0623febc151" Jul 19 08:28:55.758001 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:55.757911 2574 generic.go:356] "Generic (PLEG): container finished" podID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerID="015e968d21d414c55794891963632da2329041c8abcce8147ee46f3dfa8b98d6" exitCode=6 Jul 19 08:28:55.758001 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:55.757978 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" event={"ID":"badd07a9-342e-4d37-b2e2-54cb8b7b8d82","Type":"ContainerDied","Data":"015e968d21d414c55794891963632da2329041c8abcce8147ee46f3dfa8b98d6"} Jul 19 08:28:55.758458 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:55.758038 2574 scope.go:117] "RemoveContainer" containerID="507255c5eec000688ca39fc3729a55fa63e3949272947e782066e0623febc151" Jul 19 08:28:55.758458 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:55.758270 2574 scope.go:117] "RemoveContainer" containerID="015e968d21d414c55794891963632da2329041c8abcce8147ee46f3dfa8b98d6" Jul 19 08:28:55.758566 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:28:55.758486 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n_odh-ai-gateway-infra(badd07a9-342e-4d37-b2e2-54cb8b7b8d82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" Jul 19 08:28:56.782690 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:56.782643 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n"] Jul 19 08:28:56.914970 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:56.914946 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:57.040498 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.040397 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badd07a9-342e-4d37-b2e2-54cb8b7b8d82-maas-api-service-ca\") pod \"badd07a9-342e-4d37-b2e2-54cb8b7b8d82\" (UID: \"badd07a9-342e-4d37-b2e2-54cb8b7b8d82\") " Jul 19 08:28:57.040805 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.040784 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badd07a9-342e-4d37-b2e2-54cb8b7b8d82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "badd07a9-342e-4d37-b2e2-54cb8b7b8d82" (UID: "badd07a9-342e-4d37-b2e2-54cb8b7b8d82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:28:57.141364 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.141335 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/badd07a9-342e-4d37-b2e2-54cb8b7b8d82-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:28:57.769873 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.769845 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" Jul 19 08:28:57.770070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.769877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n" event={"ID":"badd07a9-342e-4d37-b2e2-54cb8b7b8d82","Type":"ContainerDied","Data":"eab5bd5a13615a06c94708f6a48e24bf18d899a855e2499bdf5fb39a6accbeea"} Jul 19 08:28:57.770070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.769933 2574 scope.go:117] "RemoveContainer" containerID="015e968d21d414c55794891963632da2329041c8abcce8147ee46f3dfa8b98d6" Jul 19 08:28:57.789232 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.789207 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n"] Jul 19 08:28:57.798763 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:57.798737 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qrk5n"] Jul 19 08:28:58.732568 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:58.732535 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" path="/var/lib/kubelet/pods/badd07a9-342e-4d37-b2e2-54cb8b7b8d82/volumes" Jul 19 08:28:59.726424 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:28:59.726388 2574 scope.go:117] "RemoveContainer" containerID="49f3f1b313aaa3b82a184ceadcf6ad167156b19d3eec9d539cd34841c6e14c70" Jul 19 08:29:00.785961 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:00.785925 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerID="414e1dab9509704c137fc78bb1dc9c0a3ec0aeae3b14b6d6c793f0c1b9e27613" exitCode=6 Jul 19 08:29:00.786338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:00.785980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" event={"ID":"1e34ae28-4fad-45b0-8f1a-2a0c429a0084","Type":"ContainerDied","Data":"414e1dab9509704c137fc78bb1dc9c0a3ec0aeae3b14b6d6c793f0c1b9e27613"} Jul 19 08:29:00.786338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:00.786031 2574 scope.go:117] "RemoveContainer" containerID="49f3f1b313aaa3b82a184ceadcf6ad167156b19d3eec9d539cd34841c6e14c70" Jul 19 08:29:00.786460 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:00.786372 2574 scope.go:117] "RemoveContainer" containerID="414e1dab9509704c137fc78bb1dc9c0a3ec0aeae3b14b6d6c793f0c1b9e27613" Jul 19 08:29:00.786644 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:00.786615 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9_odh-ai-gateway-infra(1e34ae28-4fad-45b0-8f1a-2a0c429a0084)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" Jul 19 08:29:01.809893 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:01.809855 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9"] Jul 19 08:29:01.948149 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:01.948121 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:29:02.088281 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.088244 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e34ae28-4fad-45b0-8f1a-2a0c429a0084-maas-api-service-ca\") pod \"1e34ae28-4fad-45b0-8f1a-2a0c429a0084\" (UID: \"1e34ae28-4fad-45b0-8f1a-2a0c429a0084\") " Jul 19 08:29:02.088600 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.088578 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e34ae28-4fad-45b0-8f1a-2a0c429a0084-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e34ae28-4fad-45b0-8f1a-2a0c429a0084" (UID: "1e34ae28-4fad-45b0-8f1a-2a0c429a0084"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:29:02.189470 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.189434 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e34ae28-4fad-45b0-8f1a-2a0c429a0084-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:29:02.798591 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.798555 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" Jul 19 08:29:02.798591 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.798573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9" event={"ID":"1e34ae28-4fad-45b0-8f1a-2a0c429a0084","Type":"ContainerDied","Data":"5becafd96daf9c9e3ecd16851e84e2edf05d1487ffb4689b73b86ead1ead4bef"} Jul 19 08:29:02.798852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.798631 2574 scope.go:117] "RemoveContainer" containerID="414e1dab9509704c137fc78bb1dc9c0a3ec0aeae3b14b6d6c793f0c1b9e27613" Jul 19 08:29:02.813805 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.813757 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9"] Jul 19 08:29:02.815375 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:02.815354 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-4pzp9"] Jul 19 08:29:04.734583 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:04.734544 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" path="/var/lib/kubelet/pods/1e34ae28-4fad-45b0-8f1a-2a0c429a0084/volumes" Jul 19 08:29:05.140472 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140438 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2"] Jul 19 08:29:05.140828 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140815 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:05.140880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140830 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:05.140880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140852 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.140880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140858 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.140880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140866 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.140880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140871 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140884 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140889 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140897 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140902 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140957 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140964 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140972 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.141034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.140980 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:05.145399 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.145381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:05.148094 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.148071 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:29:05.149348 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.149325 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2"] Jul 19 08:29:05.317614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.317574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737504ea-6f67-45b3-90e0-b13f58cd4d9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2\" (UID: \"737504ea-6f67-45b3-90e0-b13f58cd4d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:05.418496 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.418401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737504ea-6f67-45b3-90e0-b13f58cd4d9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2\" (UID: \"737504ea-6f67-45b3-90e0-b13f58cd4d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:05.419150 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.419126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737504ea-6f67-45b3-90e0-b13f58cd4d9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2\" (UID: \"737504ea-6f67-45b3-90e0-b13f58cd4d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:05.457422 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.457388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:05.604634 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.604601 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2"] Jul 19 08:29:05.607139 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:29:05.607111 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737504ea_6f67_45b3_90e0_b13f58cd4d9f.slice/crio-7452feeb1f994be7ded3cfb84916cadcaffd17600992444e27aff6dac428cd34 WatchSource:0}: Error finding container 7452feeb1f994be7ded3cfb84916cadcaffd17600992444e27aff6dac428cd34: Status 404 returned error can't find the container with id 7452feeb1f994be7ded3cfb84916cadcaffd17600992444e27aff6dac428cd34 Jul 19 08:29:05.814520 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:05.814491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" event={"ID":"737504ea-6f67-45b3-90e0-b13f58cd4d9f","Type":"ContainerStarted","Data":"7452feeb1f994be7ded3cfb84916cadcaffd17600992444e27aff6dac428cd34"} Jul 19 08:29:06.820543 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:06.820505 2574 generic.go:356] "Generic (PLEG): container finished" podID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerID="fba05401c9e2149a053dc39d2036a5bce223735762ecf247c97dabb2be468a81" exitCode=6 Jul 19 08:29:06.820941 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:06.820576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" event={"ID":"737504ea-6f67-45b3-90e0-b13f58cd4d9f","Type":"ContainerDied","Data":"fba05401c9e2149a053dc39d2036a5bce223735762ecf247c97dabb2be468a81"} Jul 19 08:29:06.820941 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:06.820798 2574 scope.go:117] "RemoveContainer" containerID="fba05401c9e2149a053dc39d2036a5bce223735762ecf247c97dabb2be468a81" Jul 19 08:29:07.827552 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:07.827457 2574 generic.go:356] "Generic (PLEG): container finished" podID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerID="c83581e3fc5d0dc420c371038790131198f0b7b113590e51a8417777ba82469b" exitCode=6 Jul 19 08:29:07.827552 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:07.827497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" event={"ID":"737504ea-6f67-45b3-90e0-b13f58cd4d9f","Type":"ContainerDied","Data":"c83581e3fc5d0dc420c371038790131198f0b7b113590e51a8417777ba82469b"} Jul 19 08:29:07.827552 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:07.827545 2574 scope.go:117] "RemoveContainer" containerID="fba05401c9e2149a053dc39d2036a5bce223735762ecf247c97dabb2be468a81" Jul 19 08:29:07.828091 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:07.827767 2574 scope.go:117] "RemoveContainer" containerID="c83581e3fc5d0dc420c371038790131198f0b7b113590e51a8417777ba82469b" Jul 19 08:29:07.828091 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:07.827991 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2_odh-ai-gateway-infra(737504ea-6f67-45b3-90e0-b13f58cd4d9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" Jul 19 08:29:08.833660 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:08.833632 2574 scope.go:117] "RemoveContainer" containerID="c83581e3fc5d0dc420c371038790131198f0b7b113590e51a8417777ba82469b" Jul 19 08:29:08.834088 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:08.833869 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2_odh-ai-gateway-infra(737504ea-6f67-45b3-90e0-b13f58cd4d9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" Jul 19 08:29:20.216478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.216442 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr"] Jul 19 08:29:20.216985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.216820 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:20.216985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.216831 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:20.216985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.216902 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e34ae28-4fad-45b0-8f1a-2a0c429a0084" containerName="revoke-keys" Jul 19 08:29:20.216985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.216912 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="badd07a9-342e-4d37-b2e2-54cb8b7b8d82" containerName="revoke-keys" Jul 19 08:29:20.220104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.220088 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:20.226236 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.226158 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr"] Jul 19 08:29:20.246619 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.246582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453df08-d274-455e-a419-0bf99eee9e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr\" (UID: \"c453df08-d274-455e-a419-0bf99eee9e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:20.347589 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.347548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453df08-d274-455e-a419-0bf99eee9e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr\" (UID: \"c453df08-d274-455e-a419-0bf99eee9e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:20.348203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.348183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453df08-d274-455e-a419-0bf99eee9e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr\" (UID: \"c453df08-d274-455e-a419-0bf99eee9e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:20.532126 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.532022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:20.667188 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.667125 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr"] Jul 19 08:29:20.669479 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:29:20.669449 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc453df08_d274_455e_a419_0bf99eee9e10.slice/crio-e78aa59ec839c7c390f1336d6f62aaf134b802750831bdf1e052f5eb02b4fcf8 WatchSource:0}: Error finding container e78aa59ec839c7c390f1336d6f62aaf134b802750831bdf1e052f5eb02b4fcf8: Status 404 returned error can't find the container with id e78aa59ec839c7c390f1336d6f62aaf134b802750831bdf1e052f5eb02b4fcf8 Jul 19 08:29:20.891309 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:20.891278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" event={"ID":"c453df08-d274-455e-a419-0bf99eee9e10","Type":"ContainerStarted","Data":"e78aa59ec839c7c390f1336d6f62aaf134b802750831bdf1e052f5eb02b4fcf8"} Jul 19 08:29:21.899037 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:21.899001 2574 generic.go:356] "Generic (PLEG): container finished" podID="c453df08-d274-455e-a419-0bf99eee9e10" containerID="132f5cf4be8d061affd3bafcab5105a9263034aedd6d5287be7d01a6fce7a187" exitCode=6 Jul 19 08:29:21.899467 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:21.899051 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" event={"ID":"c453df08-d274-455e-a419-0bf99eee9e10","Type":"ContainerDied","Data":"132f5cf4be8d061affd3bafcab5105a9263034aedd6d5287be7d01a6fce7a187"} Jul 19 08:29:21.899467 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:21.899314 2574 scope.go:117] "RemoveContainer" containerID="132f5cf4be8d061affd3bafcab5105a9263034aedd6d5287be7d01a6fce7a187" Jul 19 08:29:22.728727 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:22.728693 2574 scope.go:117] "RemoveContainer" containerID="c83581e3fc5d0dc420c371038790131198f0b7b113590e51a8417777ba82469b" Jul 19 08:29:22.905782 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:22.905751 2574 generic.go:356] "Generic (PLEG): container finished" podID="c453df08-d274-455e-a419-0bf99eee9e10" containerID="c457cd9657320517471746bfa993a3b0af0d5c7dcca9ef76b283368678a70e36" exitCode=6 Jul 19 08:29:22.906225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:22.905789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" event={"ID":"c453df08-d274-455e-a419-0bf99eee9e10","Type":"ContainerDied","Data":"c457cd9657320517471746bfa993a3b0af0d5c7dcca9ef76b283368678a70e36"} Jul 19 08:29:22.906225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:22.905850 2574 scope.go:117] "RemoveContainer" containerID="132f5cf4be8d061affd3bafcab5105a9263034aedd6d5287be7d01a6fce7a187" Jul 19 08:29:22.906225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:22.906083 2574 scope.go:117] "RemoveContainer" containerID="c457cd9657320517471746bfa993a3b0af0d5c7dcca9ef76b283368678a70e36" Jul 19 08:29:22.906387 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:22.906332 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr_odh-ai-gateway-infra(c453df08-d274-455e-a419-0bf99eee9e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" podUID="c453df08-d274-455e-a419-0bf99eee9e10" Jul 19 08:29:23.913157 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:23.913121 2574 generic.go:356] "Generic (PLEG): container finished" podID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerID="8401f35f5cfccb41add29496b0722f0644e8616c0bf052199bc4ff0cea121b52" exitCode=6 Jul 19 08:29:23.913584 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:23.913190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" event={"ID":"737504ea-6f67-45b3-90e0-b13f58cd4d9f","Type":"ContainerDied","Data":"8401f35f5cfccb41add29496b0722f0644e8616c0bf052199bc4ff0cea121b52"} Jul 19 08:29:23.913584 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:23.913258 2574 scope.go:117] "RemoveContainer" containerID="c83581e3fc5d0dc420c371038790131198f0b7b113590e51a8417777ba82469b" Jul 19 08:29:23.913783 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:23.913617 2574 scope.go:117] "RemoveContainer" containerID="8401f35f5cfccb41add29496b0722f0644e8616c0bf052199bc4ff0cea121b52" Jul 19 08:29:23.913920 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:23.913892 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2_odh-ai-gateway-infra(737504ea-6f67-45b3-90e0-b13f58cd4d9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" Jul 19 08:29:23.915167 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:23.915149 2574 scope.go:117] "RemoveContainer" containerID="c457cd9657320517471746bfa993a3b0af0d5c7dcca9ef76b283368678a70e36" Jul 19 08:29:23.915314 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:23.915298 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr_odh-ai-gateway-infra(c453df08-d274-455e-a419-0bf99eee9e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" podUID="c453df08-d274-455e-a419-0bf99eee9e10" Jul 19 08:29:24.937987 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:24.937951 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2"] Jul 19 08:29:25.089293 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.089266 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:25.191065 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.190960 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737504ea-6f67-45b3-90e0-b13f58cd4d9f-maas-api-service-ca\") pod \"737504ea-6f67-45b3-90e0-b13f58cd4d9f\" (UID: \"737504ea-6f67-45b3-90e0-b13f58cd4d9f\") " Jul 19 08:29:25.191314 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.191292 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737504ea-6f67-45b3-90e0-b13f58cd4d9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "737504ea-6f67-45b3-90e0-b13f58cd4d9f" (UID: "737504ea-6f67-45b3-90e0-b13f58cd4d9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:29:25.292658 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.292617 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737504ea-6f67-45b3-90e0-b13f58cd4d9f-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:29:25.933847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.933816 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" Jul 19 08:29:25.934060 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.933847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2" event={"ID":"737504ea-6f67-45b3-90e0-b13f58cd4d9f","Type":"ContainerDied","Data":"7452feeb1f994be7ded3cfb84916cadcaffd17600992444e27aff6dac428cd34"} Jul 19 08:29:25.934060 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.933904 2574 scope.go:117] "RemoveContainer" containerID="8401f35f5cfccb41add29496b0722f0644e8616c0bf052199bc4ff0cea121b52" Jul 19 08:29:25.956464 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.956430 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2"] Jul 19 08:29:25.962113 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:25.962077 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-rdqw2"] Jul 19 08:29:26.732695 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:26.732655 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" path="/var/lib/kubelet/pods/737504ea-6f67-45b3-90e0-b13f58cd4d9f/volumes" Jul 19 08:29:33.413950 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.413911 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt"] Jul 19 08:29:33.414338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.414301 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:29:33.414338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.414313 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:29:33.414410 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.414348 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:29:33.414410 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.414355 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:29:33.414410 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.414405 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:29:33.414509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.414413 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:29:33.418756 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.418734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:33.423221 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.423195 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt"] Jul 19 08:29:33.569070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.569028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2cb91e-1860-462c-9d1f-87d68789bec2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt\" (UID: \"0f2cb91e-1860-462c-9d1f-87d68789bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:33.669882 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.669779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2cb91e-1860-462c-9d1f-87d68789bec2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt\" (UID: \"0f2cb91e-1860-462c-9d1f-87d68789bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:33.670434 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.670414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2cb91e-1860-462c-9d1f-87d68789bec2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt\" (UID: \"0f2cb91e-1860-462c-9d1f-87d68789bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:33.729985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.729957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:33.864070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.863967 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt"] Jul 19 08:29:33.866338 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:29:33.866308 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2cb91e_1860_462c_9d1f_87d68789bec2.slice/crio-f20b593fa76cd7c9121d3aa1ffede4c34cbc03d4588bb342cfb69b05fa9fb322 WatchSource:0}: Error finding container f20b593fa76cd7c9121d3aa1ffede4c34cbc03d4588bb342cfb69b05fa9fb322: Status 404 returned error can't find the container with id f20b593fa76cd7c9121d3aa1ffede4c34cbc03d4588bb342cfb69b05fa9fb322 Jul 19 08:29:33.982287 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:33.982200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" event={"ID":"0f2cb91e-1860-462c-9d1f-87d68789bec2","Type":"ContainerStarted","Data":"f20b593fa76cd7c9121d3aa1ffede4c34cbc03d4588bb342cfb69b05fa9fb322"} Jul 19 08:29:34.988539 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:34.988503 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerID="2f872009f0c297dd9c4a8e6d2d6687111fa128b103d90de58d926dfd3b8d2f59" exitCode=6 Jul 19 08:29:34.988947 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:34.988592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" event={"ID":"0f2cb91e-1860-462c-9d1f-87d68789bec2","Type":"ContainerDied","Data":"2f872009f0c297dd9c4a8e6d2d6687111fa128b103d90de58d926dfd3b8d2f59"} Jul 19 08:29:34.988947 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:34.988761 2574 scope.go:117] "RemoveContainer" containerID="2f872009f0c297dd9c4a8e6d2d6687111fa128b103d90de58d926dfd3b8d2f59" Jul 19 08:29:35.995547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:35.995514 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerID="414c06d57f84e5a7d3192d94efbff15744fa5aab883de518210dd1e1a52e9f7d" exitCode=6 Jul 19 08:29:35.996055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:35.995602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" event={"ID":"0f2cb91e-1860-462c-9d1f-87d68789bec2","Type":"ContainerDied","Data":"414c06d57f84e5a7d3192d94efbff15744fa5aab883de518210dd1e1a52e9f7d"} Jul 19 08:29:35.996055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:35.995671 2574 scope.go:117] "RemoveContainer" containerID="2f872009f0c297dd9c4a8e6d2d6687111fa128b103d90de58d926dfd3b8d2f59" Jul 19 08:29:35.996055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:35.995811 2574 scope.go:117] "RemoveContainer" containerID="414c06d57f84e5a7d3192d94efbff15744fa5aab883de518210dd1e1a52e9f7d" Jul 19 08:29:35.996055 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:35.996031 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt_odh-ai-gateway-infra(0f2cb91e-1860-462c-9d1f-87d68789bec2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" Jul 19 08:29:37.002041 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:37.002009 2574 scope.go:117] "RemoveContainer" containerID="414c06d57f84e5a7d3192d94efbff15744fa5aab883de518210dd1e1a52e9f7d" Jul 19 08:29:37.002425 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:37.002221 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt_odh-ai-gateway-infra(0f2cb91e-1860-462c-9d1f-87d68789bec2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" Jul 19 08:29:38.726555 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:38.726518 2574 scope.go:117] "RemoveContainer" containerID="c457cd9657320517471746bfa993a3b0af0d5c7dcca9ef76b283368678a70e36" Jul 19 08:29:39.013962 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:39.013924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" event={"ID":"c453df08-d274-455e-a419-0bf99eee9e10","Type":"ContainerStarted","Data":"96bb3c77bcd69e4dbdd3b62831336842d7a60f7eb1e42e25fadb92f9e946af71"} Jul 19 08:29:39.029556 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:39.029506 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" podStartSLOduration=19.029489926 podStartE2EDuration="19.029489926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:29:39.029017167 +0000 UTC m=+2826.808680481" watchObservedRunningTime="2026-07-19 08:29:39.029489926 +0000 UTC m=+2826.809153239" Jul 19 08:29:40.020609 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:40.020564 2574 generic.go:356] "Generic (PLEG): container finished" podID="c453df08-d274-455e-a419-0bf99eee9e10" containerID="96bb3c77bcd69e4dbdd3b62831336842d7a60f7eb1e42e25fadb92f9e946af71" exitCode=6 Jul 19 08:29:40.021056 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:40.020637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" event={"ID":"c453df08-d274-455e-a419-0bf99eee9e10","Type":"ContainerDied","Data":"96bb3c77bcd69e4dbdd3b62831336842d7a60f7eb1e42e25fadb92f9e946af71"} Jul 19 08:29:40.021056 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:40.020723 2574 scope.go:117] "RemoveContainer" containerID="c457cd9657320517471746bfa993a3b0af0d5c7dcca9ef76b283368678a70e36" Jul 19 08:29:40.021056 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:40.021036 2574 scope.go:117] "RemoveContainer" containerID="96bb3c77bcd69e4dbdd3b62831336842d7a60f7eb1e42e25fadb92f9e946af71" Jul 19 08:29:40.021291 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:40.021267 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr_odh-ai-gateway-infra(c453df08-d274-455e-a419-0bf99eee9e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" podUID="c453df08-d274-455e-a419-0bf99eee9e10" Jul 19 08:29:40.045427 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:40.045389 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr"] Jul 19 08:29:41.166513 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:41.166484 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:41.240277 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:41.240236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453df08-d274-455e-a419-0bf99eee9e10-maas-api-service-ca\") pod \"c453df08-d274-455e-a419-0bf99eee9e10\" (UID: \"c453df08-d274-455e-a419-0bf99eee9e10\") " Jul 19 08:29:41.240664 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:41.240637 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c453df08-d274-455e-a419-0bf99eee9e10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c453df08-d274-455e-a419-0bf99eee9e10" (UID: "c453df08-d274-455e-a419-0bf99eee9e10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:29:41.341484 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:41.341445 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c453df08-d274-455e-a419-0bf99eee9e10-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:29:42.032690 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:42.032643 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" Jul 19 08:29:42.032690 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:42.032650 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr" event={"ID":"c453df08-d274-455e-a419-0bf99eee9e10","Type":"ContainerDied","Data":"e78aa59ec839c7c390f1336d6f62aaf134b802750831bdf1e052f5eb02b4fcf8"} Jul 19 08:29:42.032919 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:42.032738 2574 scope.go:117] "RemoveContainer" containerID="96bb3c77bcd69e4dbdd3b62831336842d7a60f7eb1e42e25fadb92f9e946af71" Jul 19 08:29:42.053304 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:42.053276 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr"] Jul 19 08:29:42.058637 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:42.058607 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-9s9gr"] Jul 19 08:29:42.732192 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:42.732156 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c453df08-d274-455e-a419-0bf99eee9e10" path="/var/lib/kubelet/pods/c453df08-d274-455e-a419-0bf99eee9e10/volumes" Jul 19 08:29:49.726163 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:49.726133 2574 scope.go:117] "RemoveContainer" containerID="414c06d57f84e5a7d3192d94efbff15744fa5aab883de518210dd1e1a52e9f7d" Jul 19 08:29:50.071243 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:50.071200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" event={"ID":"0f2cb91e-1860-462c-9d1f-87d68789bec2","Type":"ContainerStarted","Data":"4a8924f0d4bec0a56ceee849451683ebdb1021fb1fbb9666df67bb5912075a39"} Jul 19 08:29:50.087340 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:50.087291 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" podStartSLOduration=17.087275601 podStartE2EDuration="17.087275601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:29:50.085694243 +0000 UTC m=+2837.865357556" watchObservedRunningTime="2026-07-19 08:29:50.087275601 +0000 UTC m=+2837.866938915" Jul 19 08:29:51.077711 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:51.077660 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerID="4a8924f0d4bec0a56ceee849451683ebdb1021fb1fbb9666df67bb5912075a39" exitCode=6 Jul 19 08:29:51.078291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:51.077734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" event={"ID":"0f2cb91e-1860-462c-9d1f-87d68789bec2","Type":"ContainerDied","Data":"4a8924f0d4bec0a56ceee849451683ebdb1021fb1fbb9666df67bb5912075a39"} Jul 19 08:29:51.078291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:51.077792 2574 scope.go:117] "RemoveContainer" containerID="414c06d57f84e5a7d3192d94efbff15744fa5aab883de518210dd1e1a52e9f7d" Jul 19 08:29:51.078291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:51.078147 2574 scope.go:117] "RemoveContainer" containerID="4a8924f0d4bec0a56ceee849451683ebdb1021fb1fbb9666df67bb5912075a39" Jul 19 08:29:51.078461 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:29:51.078399 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt_odh-ai-gateway-infra(0f2cb91e-1860-462c-9d1f-87d68789bec2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" Jul 19 08:29:51.100403 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:51.100375 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt"] Jul 19 08:29:52.223358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:52.223331 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:52.342297 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:52.342187 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2cb91e-1860-462c-9d1f-87d68789bec2-maas-api-service-ca\") pod \"0f2cb91e-1860-462c-9d1f-87d68789bec2\" (UID: \"0f2cb91e-1860-462c-9d1f-87d68789bec2\") " Jul 19 08:29:52.342570 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:52.342547 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2cb91e-1860-462c-9d1f-87d68789bec2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f2cb91e-1860-462c-9d1f-87d68789bec2" (UID: "0f2cb91e-1860-462c-9d1f-87d68789bec2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:29:52.443539 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:52.443486 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2cb91e-1860-462c-9d1f-87d68789bec2-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:29:53.089245 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:53.089217 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" Jul 19 08:29:53.089431 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:53.089256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt" event={"ID":"0f2cb91e-1860-462c-9d1f-87d68789bec2","Type":"ContainerDied","Data":"f20b593fa76cd7c9121d3aa1ffede4c34cbc03d4588bb342cfb69b05fa9fb322"} Jul 19 08:29:53.089431 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:53.089315 2574 scope.go:117] "RemoveContainer" containerID="4a8924f0d4bec0a56ceee849451683ebdb1021fb1fbb9666df67bb5912075a39" Jul 19 08:29:53.104116 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:53.104092 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt"] Jul 19 08:29:53.109614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:53.109590 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-x57xt"] Jul 19 08:29:54.732426 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:29:54.732386 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" path="/var/lib/kubelet/pods/0f2cb91e-1860-462c-9d1f-87d68789bec2/volumes" Jul 19 08:30:00.132950 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.132920 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99"] Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133273 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133284 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133291 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133297 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133311 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133317 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133326 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133331 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:30:00.133342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133343 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:00.133621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133349 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:00.133621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133403 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:00.133621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133410 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:00.133621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133436 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:00.133621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133441 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="737504ea-6f67-45b3-90e0-b13f58cd4d9f" containerName="revoke-keys" Jul 19 08:30:00.133621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.133450 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:00.137708 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.137670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" Jul 19 08:30:00.140480 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.140455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:30:00.142283 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.142259 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99"] Jul 19 08:30:00.149911 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.149891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" Jul 19 08:30:00.282596 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:00.282564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99"] Jul 19 08:30:00.285598 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:30:00.285570 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e09ef39_3cef_4407_b9b5_fd5db4ddd710.slice/crio-05c1282e3448c4685e7370f7dd0ed75193eedbe4c8a97e7f6a023021fb4ededf WatchSource:0}: Error finding container 05c1282e3448c4685e7370f7dd0ed75193eedbe4c8a97e7f6a023021fb4ededf: Status 404 returned error can't find the container with id 05c1282e3448c4685e7370f7dd0ed75193eedbe4c8a97e7f6a023021fb4ededf Jul 19 08:30:01.125834 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:01.125792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" event={"ID":"7e09ef39-3cef-4407-b9b5-fd5db4ddd710","Type":"ContainerStarted","Data":"5439ebc830870b263adffaefd7cdc56510acf9381c8b50b151400242a1dddc8f"} Jul 19 08:30:01.126047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:01.125855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" event={"ID":"7e09ef39-3cef-4407-b9b5-fd5db4ddd710","Type":"ContainerStarted","Data":"05c1282e3448c4685e7370f7dd0ed75193eedbe4c8a97e7f6a023021fb4ededf"} Jul 19 08:30:01.140404 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:01.140349 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" podStartSLOduration=1.140334428 podStartE2EDuration="1.140334428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:01.138740601 +0000 UTC m=+2848.918403908" watchObservedRunningTime="2026-07-19 08:30:01.140334428 +0000 UTC m=+2848.919997743" Jul 19 08:30:02.132227 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:02.132191 2574 generic.go:356] "Generic (PLEG): container finished" podID="7e09ef39-3cef-4407-b9b5-fd5db4ddd710" containerID="5439ebc830870b263adffaefd7cdc56510acf9381c8b50b151400242a1dddc8f" exitCode=0 Jul 19 08:30:02.132423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:02.132285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" event={"ID":"7e09ef39-3cef-4407-b9b5-fd5db4ddd710","Type":"ContainerDied","Data":"5439ebc830870b263adffaefd7cdc56510acf9381c8b50b151400242a1dddc8f"} Jul 19 08:30:03.269906 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:03.269875 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" Jul 19 08:30:04.142474 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:04.142439 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" event={"ID":"7e09ef39-3cef-4407-b9b5-fd5db4ddd710","Type":"ContainerDied","Data":"05c1282e3448c4685e7370f7dd0ed75193eedbe4c8a97e7f6a023021fb4ededf"} Jul 19 08:30:04.142474 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:04.142478 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740830-xjv99" Jul 19 08:30:04.142744 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:04.142496 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c1282e3448c4685e7370f7dd0ed75193eedbe4c8a97e7f6a023021fb4ededf" Jul 19 08:30:05.190826 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.190787 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws"] Jul 19 08:30:05.191224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191164 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:05.191224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191176 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:05.191224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191188 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:05.191224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191193 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:05.191224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191205 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e09ef39-3cef-4407-b9b5-fd5db4ddd710" containerName="cleanup" Jul 19 08:30:05.191224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191211 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e09ef39-3cef-4407-b9b5-fd5db4ddd710" containerName="cleanup" Jul 19 08:30:05.191420 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191259 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e09ef39-3cef-4407-b9b5-fd5db4ddd710" containerName="cleanup" Jul 19 08:30:05.191420 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191267 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2cb91e-1860-462c-9d1f-87d68789bec2" containerName="revoke-keys" Jul 19 08:30:05.191420 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.191274 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c453df08-d274-455e-a419-0bf99eee9e10" containerName="revoke-keys" Jul 19 08:30:05.194251 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.194230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:05.196483 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.196464 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:30:05.201912 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.201655 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws"] Jul 19 08:30:05.255243 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.255206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-fskws\" (UID: \"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:05.355895 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.355855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-fskws\" (UID: \"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:05.356608 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.356588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-fskws\" (UID: \"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:05.505719 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.505601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:05.641588 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:05.641561 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws"] Jul 19 08:30:05.643722 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:30:05.643691 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf08e1b_096a_4b40_b3c2_fea14f3dcf5a.slice/crio-3af1acc036a09bf43ed841e841bf3a480c17fed77936ba6f04ff7d29480ddbf1 WatchSource:0}: Error finding container 3af1acc036a09bf43ed841e841bf3a480c17fed77936ba6f04ff7d29480ddbf1: Status 404 returned error can't find the container with id 3af1acc036a09bf43ed841e841bf3a480c17fed77936ba6f04ff7d29480ddbf1 Jul 19 08:30:06.152567 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:06.152529 2574 generic.go:356] "Generic (PLEG): container finished" podID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerID="40d1d98ce8e4b2b572489bbae0003fa1b6c464c083d36734df8e44f223e0fc71" exitCode=6 Jul 19 08:30:06.152787 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:06.152625 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" event={"ID":"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a","Type":"ContainerDied","Data":"40d1d98ce8e4b2b572489bbae0003fa1b6c464c083d36734df8e44f223e0fc71"} Jul 19 08:30:06.152787 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:06.152715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" event={"ID":"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a","Type":"ContainerStarted","Data":"3af1acc036a09bf43ed841e841bf3a480c17fed77936ba6f04ff7d29480ddbf1"} Jul 19 08:30:06.152890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:06.152812 2574 scope.go:117] "RemoveContainer" containerID="40d1d98ce8e4b2b572489bbae0003fa1b6c464c083d36734df8e44f223e0fc71" Jul 19 08:30:07.158856 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:07.158815 2574 generic.go:356] "Generic (PLEG): container finished" podID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerID="38dd66872192dba1c79a9b0d9919989c391b1f1f865243702c25ce079ea74f07" exitCode=6 Jul 19 08:30:07.159461 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:07.158884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" event={"ID":"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a","Type":"ContainerDied","Data":"38dd66872192dba1c79a9b0d9919989c391b1f1f865243702c25ce079ea74f07"} Jul 19 08:30:07.159461 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:07.158932 2574 scope.go:117] "RemoveContainer" containerID="40d1d98ce8e4b2b572489bbae0003fa1b6c464c083d36734df8e44f223e0fc71" Jul 19 08:30:07.159461 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:07.159181 2574 scope.go:117] "RemoveContainer" containerID="38dd66872192dba1c79a9b0d9919989c391b1f1f865243702c25ce079ea74f07" Jul 19 08:30:07.159461 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:07.159427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-fskws_odh-ai-gateway-infra(fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" Jul 19 08:30:08.166171 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:08.166145 2574 scope.go:117] "RemoveContainer" containerID="38dd66872192dba1c79a9b0d9919989c391b1f1f865243702c25ce079ea74f07" Jul 19 08:30:08.166571 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:08.166368 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-fskws_odh-ai-gateway-infra(fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" Jul 19 08:30:20.261334 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.261299 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6"] Jul 19 08:30:20.264720 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.264702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:20.272448 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.272422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6"] Jul 19 08:30:20.404629 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.404585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b65e9d-e8d1-4af2-821b-e53384fa6503-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6\" (UID: \"36b65e9d-e8d1-4af2-821b-e53384fa6503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:20.505504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.505464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b65e9d-e8d1-4af2-821b-e53384fa6503-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6\" (UID: \"36b65e9d-e8d1-4af2-821b-e53384fa6503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:20.506168 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.506141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b65e9d-e8d1-4af2-821b-e53384fa6503-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6\" (UID: \"36b65e9d-e8d1-4af2-821b-e53384fa6503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:20.576927 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.576843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:20.711590 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:20.711464 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6"] Jul 19 08:30:20.714317 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:30:20.714291 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b65e9d_e8d1_4af2_821b_e53384fa6503.slice/crio-cd2825bca4cb5ca763778cf4adccf3b469004931057f67f9b25d6c857bd71311 WatchSource:0}: Error finding container cd2825bca4cb5ca763778cf4adccf3b469004931057f67f9b25d6c857bd71311: Status 404 returned error can't find the container with id cd2825bca4cb5ca763778cf4adccf3b469004931057f67f9b25d6c857bd71311 Jul 19 08:30:21.233925 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:21.233833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" event={"ID":"36b65e9d-e8d1-4af2-821b-e53384fa6503","Type":"ContainerStarted","Data":"78b8203ac7185a1f0ace1746dbd0297084c21a76997b974d8b9a6aa5733f21dc"} Jul 19 08:30:21.233925 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:21.233891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" event={"ID":"36b65e9d-e8d1-4af2-821b-e53384fa6503","Type":"ContainerStarted","Data":"cd2825bca4cb5ca763778cf4adccf3b469004931057f67f9b25d6c857bd71311"} Jul 19 08:30:21.248480 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:21.248427 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" podStartSLOduration=1.24841065 podStartE2EDuration="1.24841065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:21.247193891 +0000 UTC m=+2869.026857205" watchObservedRunningTime="2026-07-19 08:30:21.24841065 +0000 UTC m=+2869.028073963" Jul 19 08:30:21.726064 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:21.726030 2574 scope.go:117] "RemoveContainer" containerID="38dd66872192dba1c79a9b0d9919989c391b1f1f865243702c25ce079ea74f07" Jul 19 08:30:22.241123 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:22.241087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" event={"ID":"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a","Type":"ContainerStarted","Data":"581ca54a04bddd0ccb85dc90d51a27f7c8abeaa0cc594579d1ff2ef353c0af89"} Jul 19 08:30:22.242590 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:22.242551 2574 generic.go:356] "Generic (PLEG): container finished" podID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerID="78b8203ac7185a1f0ace1746dbd0297084c21a76997b974d8b9a6aa5733f21dc" exitCode=6 Jul 19 08:30:22.242745 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:22.242626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" event={"ID":"36b65e9d-e8d1-4af2-821b-e53384fa6503","Type":"ContainerDied","Data":"78b8203ac7185a1f0ace1746dbd0297084c21a76997b974d8b9a6aa5733f21dc"} Jul 19 08:30:22.242826 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:22.242813 2574 scope.go:117] "RemoveContainer" containerID="78b8203ac7185a1f0ace1746dbd0297084c21a76997b974d8b9a6aa5733f21dc" Jul 19 08:30:22.257656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:22.257602 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" podStartSLOduration=17.257585105 podStartE2EDuration="17.257585105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:22.255917947 +0000 UTC m=+2870.035581262" watchObservedRunningTime="2026-07-19 08:30:22.257585105 +0000 UTC m=+2870.037248418" Jul 19 08:30:23.248760 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.248724 2574 generic.go:356] "Generic (PLEG): container finished" podID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerID="cd7751bd9940e0b8c7643c5ab579d4cfec57fbc3b5e09341f65468fbd1c02f9b" exitCode=6 Jul 19 08:30:23.249266 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.248816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" event={"ID":"36b65e9d-e8d1-4af2-821b-e53384fa6503","Type":"ContainerDied","Data":"cd7751bd9940e0b8c7643c5ab579d4cfec57fbc3b5e09341f65468fbd1c02f9b"} Jul 19 08:30:23.249266 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.248891 2574 scope.go:117] "RemoveContainer" containerID="78b8203ac7185a1f0ace1746dbd0297084c21a76997b974d8b9a6aa5733f21dc" Jul 19 08:30:23.249266 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.249061 2574 scope.go:117] "RemoveContainer" containerID="cd7751bd9940e0b8c7643c5ab579d4cfec57fbc3b5e09341f65468fbd1c02f9b" Jul 19 08:30:23.249425 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:23.249299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6_odh-ai-gateway-infra(36b65e9d-e8d1-4af2-821b-e53384fa6503)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" Jul 19 08:30:23.250788 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.250767 2574 generic.go:356] "Generic (PLEG): container finished" podID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerID="581ca54a04bddd0ccb85dc90d51a27f7c8abeaa0cc594579d1ff2ef353c0af89" exitCode=6 Jul 19 08:30:23.250892 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.250824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" event={"ID":"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a","Type":"ContainerDied","Data":"581ca54a04bddd0ccb85dc90d51a27f7c8abeaa0cc594579d1ff2ef353c0af89"} Jul 19 08:30:23.251083 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.251068 2574 scope.go:117] "RemoveContainer" containerID="581ca54a04bddd0ccb85dc90d51a27f7c8abeaa0cc594579d1ff2ef353c0af89" Jul 19 08:30:23.251274 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:23.251258 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-fskws_odh-ai-gateway-infra(fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" Jul 19 08:30:23.260268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.260247 2574 scope.go:117] "RemoveContainer" containerID="38dd66872192dba1c79a9b0d9919989c391b1f1f865243702c25ce079ea74f07" Jul 19 08:30:23.267362 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:23.267330 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws"] Jul 19 08:30:24.265669 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:24.265648 2574 scope.go:117] "RemoveContainer" containerID="cd7751bd9940e0b8c7643c5ab579d4cfec57fbc3b5e09341f65468fbd1c02f9b" Jul 19 08:30:24.265986 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:24.265874 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6_odh-ai-gateway-infra(36b65e9d-e8d1-4af2-821b-e53384fa6503)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" Jul 19 08:30:24.398492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:24.398462 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:24.541427 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:24.541328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a-maas-api-service-ca\") pod \"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a\" (UID: \"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a\") " Jul 19 08:30:24.541790 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:24.541770 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" (UID: "fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:30:24.642618 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:24.642570 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:30:25.271566 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:25.271531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" event={"ID":"fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a","Type":"ContainerDied","Data":"3af1acc036a09bf43ed841e841bf3a480c17fed77936ba6f04ff7d29480ddbf1"} Jul 19 08:30:25.271566 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:25.271548 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws" Jul 19 08:30:25.272104 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:25.271597 2574 scope.go:117] "RemoveContainer" containerID="581ca54a04bddd0ccb85dc90d51a27f7c8abeaa0cc594579d1ff2ef353c0af89" Jul 19 08:30:25.286006 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:25.285978 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws"] Jul 19 08:30:25.291212 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:25.291189 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-fskws"] Jul 19 08:30:26.735129 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:26.735091 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" path="/var/lib/kubelet/pods/fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a/volumes" Jul 19 08:30:27.505479 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505443 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f"] Jul 19 08:30:27.505892 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505879 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.505945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505895 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.505945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505908 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.505945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505917 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.505945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505928 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.505945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505933 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.506099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.505989 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.506099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.506002 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.506099 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.506009 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf08e1b-096a-4b40-b3c2-fea14f3dcf5a" containerName="revoke-keys" Jul 19 08:30:27.510430 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.510414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:27.514760 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.514715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f"] Jul 19 08:30:27.569572 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.569531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8b041d-5cfe-4c61-816c-616abf17784b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f\" (UID: \"bd8b041d-5cfe-4c61-816c-616abf17784b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:27.670223 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.670179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8b041d-5cfe-4c61-816c-616abf17784b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f\" (UID: \"bd8b041d-5cfe-4c61-816c-616abf17784b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:27.670983 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.670963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8b041d-5cfe-4c61-816c-616abf17784b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f\" (UID: \"bd8b041d-5cfe-4c61-816c-616abf17784b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:27.821773 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.821636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:27.954358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:27.954335 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f"] Jul 19 08:30:27.956510 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:30:27.956483 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8b041d_5cfe_4c61_816c_616abf17784b.slice/crio-a50285ce6167752a32711391718ec70b9e59f1258d9d75146145215b75826bfa WatchSource:0}: Error finding container a50285ce6167752a32711391718ec70b9e59f1258d9d75146145215b75826bfa: Status 404 returned error can't find the container with id a50285ce6167752a32711391718ec70b9e59f1258d9d75146145215b75826bfa Jul 19 08:30:28.290121 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:28.290088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" event={"ID":"bd8b041d-5cfe-4c61-816c-616abf17784b","Type":"ContainerStarted","Data":"a50285ce6167752a32711391718ec70b9e59f1258d9d75146145215b75826bfa"} Jul 19 08:30:29.297136 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:29.297103 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerID="4c1921b24b2ad160972bc19bad5fc117e85daca001e0ca3ee448ed04168eb494" exitCode=6 Jul 19 08:30:29.297495 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:29.297188 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" event={"ID":"bd8b041d-5cfe-4c61-816c-616abf17784b","Type":"ContainerDied","Data":"4c1921b24b2ad160972bc19bad5fc117e85daca001e0ca3ee448ed04168eb494"} Jul 19 08:30:29.297495 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:29.297328 2574 scope.go:117] "RemoveContainer" containerID="4c1921b24b2ad160972bc19bad5fc117e85daca001e0ca3ee448ed04168eb494" Jul 19 08:30:30.304016 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:30.303917 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerID="a0ed41072524cac5e1f0fae1337714b47e58ec0295e8deaf97dd72809ca63355" exitCode=6 Jul 19 08:30:30.304016 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:30.303999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" event={"ID":"bd8b041d-5cfe-4c61-816c-616abf17784b","Type":"ContainerDied","Data":"a0ed41072524cac5e1f0fae1337714b47e58ec0295e8deaf97dd72809ca63355"} Jul 19 08:30:30.304532 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:30.304088 2574 scope.go:117] "RemoveContainer" containerID="4c1921b24b2ad160972bc19bad5fc117e85daca001e0ca3ee448ed04168eb494" Jul 19 08:30:30.304532 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:30.304178 2574 scope.go:117] "RemoveContainer" containerID="a0ed41072524cac5e1f0fae1337714b47e58ec0295e8deaf97dd72809ca63355" Jul 19 08:30:30.304532 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:30.304434 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f_odh-ai-gateway-infra(bd8b041d-5cfe-4c61-816c-616abf17784b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" Jul 19 08:30:31.310637 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:31.310608 2574 scope.go:117] "RemoveContainer" containerID="a0ed41072524cac5e1f0fae1337714b47e58ec0295e8deaf97dd72809ca63355" Jul 19 08:30:31.311042 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:31.310833 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f_odh-ai-gateway-infra(bd8b041d-5cfe-4c61-816c-616abf17784b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" Jul 19 08:30:33.979628 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.979595 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx"] Jul 19 08:30:33.990233 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.990194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:33.992777 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.992746 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx"] Jul 19 08:30:33.993388 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.993366 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49f9c5\"/\"test-model-49f9c5-kserve-self-signed-certs\"" Jul 19 08:30:33.993493 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.993410 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49f9c5\"/\"kube-root-ca.crt\"" Jul 19 08:30:33.993493 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.993374 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49f9c5\"/\"openshift-service-ca.crt\"" Jul 19 08:30:33.993493 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:33.993375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49f9c5\"/\"default-dockercfg-qrgxw\"" Jul 19 08:30:34.131083 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.131039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjbw\" (UniqueName: \"kubernetes.io/projected/9c5dd865-a272-4a20-b48f-2478911b655d-kube-api-access-xwjbw\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.131268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.131094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5dd865-a272-4a20-b48f-2478911b655d-tls-certs\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.131268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.131133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-dshm\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.131268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.131152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-model-cache\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.131268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.131190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-home\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.131268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.131224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-tmp-dir\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.231942 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.231845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-home\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.231942 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.231889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-tmp-dir\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.231942 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.231944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjbw\" (UniqueName: \"kubernetes.io/projected/9c5dd865-a272-4a20-b48f-2478911b655d-kube-api-access-xwjbw\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.232227 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.231985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5dd865-a272-4a20-b48f-2478911b655d-tls-certs\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.232227 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.232012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-dshm\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.232227 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.232129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-model-cache\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.232385 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.232299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-home\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.232385 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.232317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-tmp-dir\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.232462 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.232449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-model-cache\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.234448 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.234420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-dshm\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.234633 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.234618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5dd865-a272-4a20-b48f-2478911b655d-tls-certs\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.239009 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.238990 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjbw\" (UniqueName: \"kubernetes.io/projected/9c5dd865-a272-4a20-b48f-2478911b655d-kube-api-access-xwjbw\") pod \"test-model-49f9c5-kserve-7d7767c469-6k8hx\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.300163 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.300125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:34.435138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:34.435107 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx"] Jul 19 08:30:34.437116 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:30:34.437091 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c5dd865_a272_4a20_b48f_2478911b655d.slice/crio-475fb0500dcbd8c44401facedbed100cce9ec5b1d3c08efab3352e7265a395bd WatchSource:0}: Error finding container 475fb0500dcbd8c44401facedbed100cce9ec5b1d3c08efab3352e7265a395bd: Status 404 returned error can't find the container with id 475fb0500dcbd8c44401facedbed100cce9ec5b1d3c08efab3352e7265a395bd Jul 19 08:30:35.332245 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:35.332202 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" event={"ID":"9c5dd865-a272-4a20-b48f-2478911b655d","Type":"ContainerStarted","Data":"3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24"} Jul 19 08:30:35.332245 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:35.332249 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" event={"ID":"9c5dd865-a272-4a20-b48f-2478911b655d","Type":"ContainerStarted","Data":"475fb0500dcbd8c44401facedbed100cce9ec5b1d3c08efab3352e7265a395bd"} Jul 19 08:30:35.347478 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:35.347420 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" podStartSLOduration=2.347402785 podStartE2EDuration="2.347402785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:30:35.346288904 +0000 UTC m=+2883.125952223" watchObservedRunningTime="2026-07-19 08:30:35.347402785 +0000 UTC m=+2883.127066134" Jul 19 08:30:37.726426 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:37.726399 2574 scope.go:117] "RemoveContainer" containerID="cd7751bd9940e0b8c7643c5ab579d4cfec57fbc3b5e09341f65468fbd1c02f9b" Jul 19 08:30:38.347578 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:38.347539 2574 generic.go:356] "Generic (PLEG): container finished" podID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerID="d14fcdc0308d5f13f6aaa52eb9fe39d7d3bbaf06b1f2007047ea88641adcba3c" exitCode=6 Jul 19 08:30:38.347828 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:38.347610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" event={"ID":"36b65e9d-e8d1-4af2-821b-e53384fa6503","Type":"ContainerDied","Data":"d14fcdc0308d5f13f6aaa52eb9fe39d7d3bbaf06b1f2007047ea88641adcba3c"} Jul 19 08:30:38.347828 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:38.347667 2574 scope.go:117] "RemoveContainer" containerID="cd7751bd9940e0b8c7643c5ab579d4cfec57fbc3b5e09341f65468fbd1c02f9b" Jul 19 08:30:38.348029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:38.348010 2574 scope.go:117] "RemoveContainer" containerID="d14fcdc0308d5f13f6aaa52eb9fe39d7d3bbaf06b1f2007047ea88641adcba3c" Jul 19 08:30:38.348278 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:38.348250 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6_odh-ai-gateway-infra(36b65e9d-e8d1-4af2-821b-e53384fa6503)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" Jul 19 08:30:39.372532 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:39.372498 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6"] Jul 19 08:30:39.512898 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:39.512871 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:39.685337 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:39.685238 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b65e9d-e8d1-4af2-821b-e53384fa6503-maas-api-service-ca\") pod \"36b65e9d-e8d1-4af2-821b-e53384fa6503\" (UID: \"36b65e9d-e8d1-4af2-821b-e53384fa6503\") " Jul 19 08:30:39.685655 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:39.685634 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b65e9d-e8d1-4af2-821b-e53384fa6503-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36b65e9d-e8d1-4af2-821b-e53384fa6503" (UID: "36b65e9d-e8d1-4af2-821b-e53384fa6503"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:30:39.786937 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:39.786896 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36b65e9d-e8d1-4af2-821b-e53384fa6503-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:30:40.360089 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:40.360048 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" event={"ID":"36b65e9d-e8d1-4af2-821b-e53384fa6503","Type":"ContainerDied","Data":"cd2825bca4cb5ca763778cf4adccf3b469004931057f67f9b25d6c857bd71311"} Jul 19 08:30:40.360285 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:40.360148 2574 scope.go:117] "RemoveContainer" containerID="d14fcdc0308d5f13f6aaa52eb9fe39d7d3bbaf06b1f2007047ea88641adcba3c" Jul 19 08:30:40.360285 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:40.360059 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6" Jul 19 08:30:40.382214 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:40.382173 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6"] Jul 19 08:30:40.386328 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:40.386287 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tqwf6"] Jul 19 08:30:40.732352 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:40.732272 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" path="/var/lib/kubelet/pods/36b65e9d-e8d1-4af2-821b-e53384fa6503/volumes" Jul 19 08:30:44.301121 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:44.301085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:44.301523 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:44.301132 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:44.313613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:44.313586 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:44.390294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:44.390264 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:30:46.727097 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:46.727054 2574 scope.go:117] "RemoveContainer" containerID="a0ed41072524cac5e1f0fae1337714b47e58ec0295e8deaf97dd72809ca63355" Jul 19 08:30:47.395294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:47.395258 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerID="17a6bf4ebff7436f2a0d2e20f4e8a06f2241184dc9bd65c911110f4e15e44494" exitCode=6 Jul 19 08:30:47.395294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:47.395297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" event={"ID":"bd8b041d-5cfe-4c61-816c-616abf17784b","Type":"ContainerDied","Data":"17a6bf4ebff7436f2a0d2e20f4e8a06f2241184dc9bd65c911110f4e15e44494"} Jul 19 08:30:47.395554 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:47.395355 2574 scope.go:117] "RemoveContainer" containerID="a0ed41072524cac5e1f0fae1337714b47e58ec0295e8deaf97dd72809ca63355" Jul 19 08:30:47.395636 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:47.395620 2574 scope.go:117] "RemoveContainer" containerID="17a6bf4ebff7436f2a0d2e20f4e8a06f2241184dc9bd65c911110f4e15e44494" Jul 19 08:30:47.395897 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:30:47.395872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f_odh-ai-gateway-infra(bd8b041d-5cfe-4c61-816c-616abf17784b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" Jul 19 08:30:48.418986 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:48.418953 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f"] Jul 19 08:30:48.559263 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:48.559239 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:48.662765 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:48.662721 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8b041d-5cfe-4c61-816c-616abf17784b-maas-api-service-ca\") pod \"bd8b041d-5cfe-4c61-816c-616abf17784b\" (UID: \"bd8b041d-5cfe-4c61-816c-616abf17784b\") " Jul 19 08:30:48.663095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:48.663072 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8b041d-5cfe-4c61-816c-616abf17784b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd8b041d-5cfe-4c61-816c-616abf17784b" (UID: "bd8b041d-5cfe-4c61-816c-616abf17784b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:30:48.763550 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:48.763510 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8b041d-5cfe-4c61-816c-616abf17784b-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:30:49.406955 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:49.406924 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" Jul 19 08:30:49.407155 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:49.406922 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f" event={"ID":"bd8b041d-5cfe-4c61-816c-616abf17784b","Type":"ContainerDied","Data":"a50285ce6167752a32711391718ec70b9e59f1258d9d75146145215b75826bfa"} Jul 19 08:30:49.407155 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:49.407060 2574 scope.go:117] "RemoveContainer" containerID="17a6bf4ebff7436f2a0d2e20f4e8a06f2241184dc9bd65c911110f4e15e44494" Jul 19 08:30:49.421343 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:49.421312 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f"] Jul 19 08:30:49.425558 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:49.425533 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4rn6f"] Jul 19 08:30:50.731570 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:50.731537 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" path="/var/lib/kubelet/pods/bd8b041d-5cfe-4c61-816c-616abf17784b/volumes" Jul 19 08:30:59.773754 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.773722 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt"] Jul 19 08:30:59.774268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774234 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774258 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774268 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774274 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774282 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774301 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774309 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774338 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774348 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774360 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774367 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774483 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774530 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774541 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774550 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:30:59.774904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774557 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.774904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.774564 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8b041d-5cfe-4c61-816c-616abf17784b" containerName="revoke-keys" Jul 19 08:30:59.779067 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.779044 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.782280 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.782253 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-916a44\"/\"kube-root-ca.crt\"" Jul 19 08:30:59.782423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.782276 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-916a44\"/\"openshift-service-ca.crt\"" Jul 19 08:30:59.782423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.782336 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-916a44\"/\"default-dockercfg-2mhks\"" Jul 19 08:30:59.782423 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.782260 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-916a44\"/\"test-model-916a44-kserve-self-signed-certs\"" Jul 19 08:30:59.785808 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.785786 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt"] Jul 19 08:30:59.866897 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.866857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-model-cache\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.866897 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.866910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-home\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.867160 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.866927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tmp-dir\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.867160 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.867008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndd8\" (UniqueName: \"kubernetes.io/projected/113f78c9-c547-4e4a-a9b6-13e14f93bd67-kube-api-access-tndd8\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.867160 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.867051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-dshm\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.867160 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.867079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tls-certs\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968262 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-model-cache\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-home\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tmp-dir\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndd8\" (UniqueName: \"kubernetes.io/projected/113f78c9-c547-4e4a-a9b6-13e14f93bd67-kube-api-access-tndd8\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-dshm\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tls-certs\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968780 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-home\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968780 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968767 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tmp-dir\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.968892 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.968822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-model-cache\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.970786 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.970760 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-dshm\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.970963 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.970947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tls-certs\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:30:59.975902 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:30:59.975885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndd8\" (UniqueName: \"kubernetes.io/projected/113f78c9-c547-4e4a-a9b6-13e14f93bd67-kube-api-access-tndd8\") pod \"test-model-916a44-kserve-7bdd68f747-jc6bt\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:00.091368 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:00.091333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:00.226628 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:00.226602 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt"] Jul 19 08:31:00.228733 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:31:00.228708 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod113f78c9_c547_4e4a_a9b6_13e14f93bd67.slice/crio-384546ad8cefe7865a606e5b034ef0966d4d3703017e83b111e553d74ca8da30 WatchSource:0}: Error finding container 384546ad8cefe7865a606e5b034ef0966d4d3703017e83b111e553d74ca8da30: Status 404 returned error can't find the container with id 384546ad8cefe7865a606e5b034ef0966d4d3703017e83b111e553d74ca8da30 Jul 19 08:31:00.463101 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:00.463014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" event={"ID":"113f78c9-c547-4e4a-a9b6-13e14f93bd67","Type":"ContainerStarted","Data":"b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef"} Jul 19 08:31:00.463101 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:00.463058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" event={"ID":"113f78c9-c547-4e4a-a9b6-13e14f93bd67","Type":"ContainerStarted","Data":"384546ad8cefe7865a606e5b034ef0966d4d3703017e83b111e553d74ca8da30"} Jul 19 08:31:00.480713 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:00.480646 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" podStartSLOduration=1.480628206 podStartE2EDuration="1.480628206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:31:00.47863074 +0000 UTC m=+2908.258294054" watchObservedRunningTime="2026-07-19 08:31:00.480628206 +0000 UTC m=+2908.260291519" Jul 19 08:31:10.092178 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:10.092134 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:10.092673 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:10.092193 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:10.104866 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:10.104841 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:10.523229 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:10.523141 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:21.278225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:21.278191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx"] Jul 19 08:31:22.193585 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:22.193553 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt"] Jul 19 08:31:36.299093 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.299051 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" podUID="9c5dd865-a272-4a20-b48f-2478911b655d" containerName="main" containerID="cri-o://3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24" gracePeriod=45 Jul 19 08:31:36.559235 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.559166 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:31:36.616005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.615970 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-dshm\") pod \"9c5dd865-a272-4a20-b48f-2478911b655d\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " Jul 19 08:31:36.616225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616087 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-tmp-dir\") pod \"9c5dd865-a272-4a20-b48f-2478911b655d\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " Jul 19 08:31:36.616225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-model-cache\") pod \"9c5dd865-a272-4a20-b48f-2478911b655d\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " Jul 19 08:31:36.616225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616140 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5dd865-a272-4a20-b48f-2478911b655d-tls-certs\") pod \"9c5dd865-a272-4a20-b48f-2478911b655d\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " Jul 19 08:31:36.616225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616191 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-home\") pod \"9c5dd865-a272-4a20-b48f-2478911b655d\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " Jul 19 08:31:36.616225 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616224 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwjbw\" (UniqueName: \"kubernetes.io/projected/9c5dd865-a272-4a20-b48f-2478911b655d-kube-api-access-xwjbw\") pod \"9c5dd865-a272-4a20-b48f-2478911b655d\" (UID: \"9c5dd865-a272-4a20-b48f-2478911b655d\") " Jul 19 08:31:36.616485 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616388 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c5dd865-a272-4a20-b48f-2478911b655d" (UID: "9c5dd865-a272-4a20-b48f-2478911b655d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:36.616485 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616399 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c5dd865-a272-4a20-b48f-2478911b655d" (UID: "9c5dd865-a272-4a20-b48f-2478911b655d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:36.616585 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616516 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-home" (OuterVolumeSpecName: "home") pod "9c5dd865-a272-4a20-b48f-2478911b655d" (UID: "9c5dd865-a272-4a20-b48f-2478911b655d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:36.616585 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616530 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:36.616585 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.616547 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:36.618456 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.618424 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-dshm" (OuterVolumeSpecName: "dshm") pod "9c5dd865-a272-4a20-b48f-2478911b655d" (UID: "9c5dd865-a272-4a20-b48f-2478911b655d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:36.618555 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.618514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5dd865-a272-4a20-b48f-2478911b655d-kube-api-access-xwjbw" (OuterVolumeSpecName: "kube-api-access-xwjbw") pod "9c5dd865-a272-4a20-b48f-2478911b655d" (UID: "9c5dd865-a272-4a20-b48f-2478911b655d"). InnerVolumeSpecName "kube-api-access-xwjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:31:36.618555 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.618524 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5dd865-a272-4a20-b48f-2478911b655d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c5dd865-a272-4a20-b48f-2478911b655d" (UID: "9c5dd865-a272-4a20-b48f-2478911b655d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:31:36.640411 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.640377 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c5dd865-a272-4a20-b48f-2478911b655d" containerID="3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24" exitCode=0 Jul 19 08:31:36.640613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.640425 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" Jul 19 08:31:36.640613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.640468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" event={"ID":"9c5dd865-a272-4a20-b48f-2478911b655d","Type":"ContainerDied","Data":"3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24"} Jul 19 08:31:36.640613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.640517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx" event={"ID":"9c5dd865-a272-4a20-b48f-2478911b655d","Type":"ContainerDied","Data":"475fb0500dcbd8c44401facedbed100cce9ec5b1d3c08efab3352e7265a395bd"} Jul 19 08:31:36.640613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.640540 2574 scope.go:117] "RemoveContainer" containerID="3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24" Jul 19 08:31:36.653144 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.651289 2574 scope.go:117] "RemoveContainer" containerID="3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24" Jul 19 08:31:36.653144 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:31:36.652528 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24\": container with ID starting with 3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24 not found: ID does not exist" containerID="3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24" Jul 19 08:31:36.653144 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.652563 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24"} err="failed to get container status \"3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24\": rpc error: code = NotFound desc = could not find container \"3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24\": container with ID starting with 3580879f86cee577e62b8f502fc132f80805b6212b8da926c634f1d240a02e24 not found: ID does not exist" Jul 19 08:31:36.666189 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.666156 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx"] Jul 19 08:31:36.669373 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.669348 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-49f9c5/test-model-49f9c5-kserve-7d7767c469-6k8hx"] Jul 19 08:31:36.717206 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.717170 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5dd865-a272-4a20-b48f-2478911b655d-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:36.717206 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.717198 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:36.717206 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.717208 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjbw\" (UniqueName: \"kubernetes.io/projected/9c5dd865-a272-4a20-b48f-2478911b655d-kube-api-access-xwjbw\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:36.717426 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.717220 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5dd865-a272-4a20-b48f-2478911b655d-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:36.731930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:36.731887 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5dd865-a272-4a20-b48f-2478911b655d" path="/var/lib/kubelet/pods/9c5dd865-a272-4a20-b48f-2478911b655d/volumes" Jul 19 08:31:37.214599 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.214557 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" podUID="113f78c9-c547-4e4a-a9b6-13e14f93bd67" containerName="main" containerID="cri-o://b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef" gracePeriod=45 Jul 19 08:31:37.476106 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.476049 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:37.524493 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524455 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-home\") pod \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " Jul 19 08:31:37.524668 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524500 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-model-cache\") pod \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " Jul 19 08:31:37.524668 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524608 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tls-certs\") pod \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " Jul 19 08:31:37.524668 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tndd8\" (UniqueName: \"kubernetes.io/projected/113f78c9-c547-4e4a-a9b6-13e14f93bd67-kube-api-access-tndd8\") pod \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " Jul 19 08:31:37.524848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524719 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tmp-dir\") pod \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " Jul 19 08:31:37.524848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524741 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-dshm\") pod \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\" (UID: \"113f78c9-c547-4e4a-a9b6-13e14f93bd67\") " Jul 19 08:31:37.524848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524791 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-model-cache" (OuterVolumeSpecName: "model-cache") pod "113f78c9-c547-4e4a-a9b6-13e14f93bd67" (UID: "113f78c9-c547-4e4a-a9b6-13e14f93bd67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:37.524848 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.524803 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-home" (OuterVolumeSpecName: "home") pod "113f78c9-c547-4e4a-a9b6-13e14f93bd67" (UID: "113f78c9-c547-4e4a-a9b6-13e14f93bd67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:37.525061 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.525004 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:37.525061 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.525019 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:37.525061 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.525033 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "113f78c9-c547-4e4a-a9b6-13e14f93bd67" (UID: "113f78c9-c547-4e4a-a9b6-13e14f93bd67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:37.527065 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.527041 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "113f78c9-c547-4e4a-a9b6-13e14f93bd67" (UID: "113f78c9-c547-4e4a-a9b6-13e14f93bd67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:31:37.527065 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.527055 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113f78c9-c547-4e4a-a9b6-13e14f93bd67-kube-api-access-tndd8" (OuterVolumeSpecName: "kube-api-access-tndd8") pod "113f78c9-c547-4e4a-a9b6-13e14f93bd67" (UID: "113f78c9-c547-4e4a-a9b6-13e14f93bd67"). InnerVolumeSpecName "kube-api-access-tndd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:31:37.527179 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.527041 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-dshm" (OuterVolumeSpecName: "dshm") pod "113f78c9-c547-4e4a-a9b6-13e14f93bd67" (UID: "113f78c9-c547-4e4a-a9b6-13e14f93bd67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:31:37.625757 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.625716 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:37.625757 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.625750 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndd8\" (UniqueName: \"kubernetes.io/projected/113f78c9-c547-4e4a-a9b6-13e14f93bd67-kube-api-access-tndd8\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:37.625975 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.625765 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:37.625975 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.625779 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f78c9-c547-4e4a-a9b6-13e14f93bd67-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:31:37.647940 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.647901 2574 generic.go:356] "Generic (PLEG): container finished" podID="113f78c9-c547-4e4a-a9b6-13e14f93bd67" containerID="b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef" exitCode=0 Jul 19 08:31:37.648109 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.647944 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" Jul 19 08:31:37.648109 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.647989 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" event={"ID":"113f78c9-c547-4e4a-a9b6-13e14f93bd67","Type":"ContainerDied","Data":"b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef"} Jul 19 08:31:37.648109 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.648026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt" event={"ID":"113f78c9-c547-4e4a-a9b6-13e14f93bd67","Type":"ContainerDied","Data":"384546ad8cefe7865a606e5b034ef0966d4d3703017e83b111e553d74ca8da30"} Jul 19 08:31:37.648109 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.648043 2574 scope.go:117] "RemoveContainer" containerID="b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef" Jul 19 08:31:37.656885 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.656864 2574 scope.go:117] "RemoveContainer" containerID="b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef" Jul 19 08:31:37.657136 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:31:37.657120 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef\": container with ID starting with b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef not found: ID does not exist" containerID="b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef" Jul 19 08:31:37.657187 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.657144 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef"} err="failed to get container status \"b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef\": rpc error: code = NotFound desc = could not find container \"b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef\": container with ID starting with b278a506959b450afe1e3b1428b55d1f02e6036c3a9931599dcb5ef5512756ef not found: ID does not exist" Jul 19 08:31:37.667906 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.667882 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt"] Jul 19 08:31:37.671502 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:37.671482 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-916a44/test-model-916a44-kserve-7bdd68f747-jc6bt"] Jul 19 08:31:38.732494 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:31:38.732456 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113f78c9-c547-4e4a-a9b6-13e14f93bd67" path="/var/lib/kubelet/pods/113f78c9-c547-4e4a-a9b6-13e14f93bd67/volumes" Jul 19 08:32:05.276114 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276024 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2"] Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276440 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5dd865-a272-4a20-b48f-2478911b655d" containerName="main" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276453 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5dd865-a272-4a20-b48f-2478911b655d" containerName="main" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276467 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276473 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b65e9d-e8d1-4af2-821b-e53384fa6503" containerName="revoke-keys" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276485 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113f78c9-c547-4e4a-a9b6-13e14f93bd67" containerName="main" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276490 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="113f78c9-c547-4e4a-a9b6-13e14f93bd67" containerName="main" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276543 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="113f78c9-c547-4e4a-a9b6-13e14f93bd67" containerName="main" Jul 19 08:32:05.276616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.276552 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5dd865-a272-4a20-b48f-2478911b655d" containerName="main" Jul 19 08:32:05.279713 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.279696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:05.281928 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.281908 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:32:05.285500 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.285474 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2"] Jul 19 08:32:05.377070 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.377030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82096131-8ebe-49b7-85a2-80b545213919-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2\" (UID: \"82096131-8ebe-49b7-85a2-80b545213919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:05.478509 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.478465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82096131-8ebe-49b7-85a2-80b545213919-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2\" (UID: \"82096131-8ebe-49b7-85a2-80b545213919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:05.479151 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.479132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82096131-8ebe-49b7-85a2-80b545213919-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2\" (UID: \"82096131-8ebe-49b7-85a2-80b545213919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:05.591450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.591420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:05.724389 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.724354 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2"] Jul 19 08:32:05.727386 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:32:05.727351 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82096131_8ebe_49b7_85a2_80b545213919.slice/crio-24afe140164ea2fc5c41f750a2a4739ff231afc7e94a036b9e2b32c8496c3b45 WatchSource:0}: Error finding container 24afe140164ea2fc5c41f750a2a4739ff231afc7e94a036b9e2b32c8496c3b45: Status 404 returned error can't find the container with id 24afe140164ea2fc5c41f750a2a4739ff231afc7e94a036b9e2b32c8496c3b45 Jul 19 08:32:05.778215 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:05.778183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" event={"ID":"82096131-8ebe-49b7-85a2-80b545213919","Type":"ContainerStarted","Data":"24afe140164ea2fc5c41f750a2a4739ff231afc7e94a036b9e2b32c8496c3b45"} Jul 19 08:32:06.784635 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:06.784601 2574 generic.go:356] "Generic (PLEG): container finished" podID="82096131-8ebe-49b7-85a2-80b545213919" containerID="48d3dd6f57c81de74cf6a3cd92ceffbc5497a4ee8514c5315d9b0c974470c5c7" exitCode=6 Jul 19 08:32:06.785057 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:06.784647 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" event={"ID":"82096131-8ebe-49b7-85a2-80b545213919","Type":"ContainerDied","Data":"48d3dd6f57c81de74cf6a3cd92ceffbc5497a4ee8514c5315d9b0c974470c5c7"} Jul 19 08:32:06.785057 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:06.784883 2574 scope.go:117] "RemoveContainer" containerID="48d3dd6f57c81de74cf6a3cd92ceffbc5497a4ee8514c5315d9b0c974470c5c7" Jul 19 08:32:07.792317 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:07.792233 2574 generic.go:356] "Generic (PLEG): container finished" podID="82096131-8ebe-49b7-85a2-80b545213919" containerID="e3336c62efd8b53dd56ad75e7191c4b89658ddb080489480a2aadef24c57df8b" exitCode=6 Jul 19 08:32:07.792774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:07.792318 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" event={"ID":"82096131-8ebe-49b7-85a2-80b545213919","Type":"ContainerDied","Data":"e3336c62efd8b53dd56ad75e7191c4b89658ddb080489480a2aadef24c57df8b"} Jul 19 08:32:07.792774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:07.792372 2574 scope.go:117] "RemoveContainer" containerID="48d3dd6f57c81de74cf6a3cd92ceffbc5497a4ee8514c5315d9b0c974470c5c7" Jul 19 08:32:07.792774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:07.792521 2574 scope.go:117] "RemoveContainer" containerID="e3336c62efd8b53dd56ad75e7191c4b89658ddb080489480a2aadef24c57df8b" Jul 19 08:32:07.792774 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:07.792758 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2_odh-ai-gateway-infra(82096131-8ebe-49b7-85a2-80b545213919)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" podUID="82096131-8ebe-49b7-85a2-80b545213919" Jul 19 08:32:08.799215 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:08.799186 2574 scope.go:117] "RemoveContainer" containerID="e3336c62efd8b53dd56ad75e7191c4b89658ddb080489480a2aadef24c57df8b" Jul 19 08:32:08.799614 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:08.799382 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2_odh-ai-gateway-infra(82096131-8ebe-49b7-85a2-80b545213919)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" podUID="82096131-8ebe-49b7-85a2-80b545213919" Jul 19 08:32:10.353169 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.353124 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q"] Jul 19 08:32:10.356933 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.356906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:10.362664 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.362634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q"] Jul 19 08:32:10.526734 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.526662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8d16c0-6637-40e0-80bf-e725642fe1f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q\" (UID: \"1f8d16c0-6637-40e0-80bf-e725642fe1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:10.628079 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.627982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8d16c0-6637-40e0-80bf-e725642fe1f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q\" (UID: \"1f8d16c0-6637-40e0-80bf-e725642fe1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:10.628765 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.628741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8d16c0-6637-40e0-80bf-e725642fe1f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q\" (UID: \"1f8d16c0-6637-40e0-80bf-e725642fe1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:10.670448 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.670410 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:10.806473 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:10.806439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q"] Jul 19 08:32:10.810177 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:32:10.810149 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8d16c0_6637_40e0_80bf_e725642fe1f3.slice/crio-66c32edfdbc151e6465794807e2c6916ed7d1b7d757475f462f4e317cda1d9f5 WatchSource:0}: Error finding container 66c32edfdbc151e6465794807e2c6916ed7d1b7d757475f462f4e317cda1d9f5: Status 404 returned error can't find the container with id 66c32edfdbc151e6465794807e2c6916ed7d1b7d757475f462f4e317cda1d9f5 Jul 19 08:32:11.815222 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:11.815187 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerID="c5bf073cc763a1d9d28d71af483483518cb28ceae5a01d25323ff33cb76acb4a" exitCode=6 Jul 19 08:32:11.815744 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:11.815244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" event={"ID":"1f8d16c0-6637-40e0-80bf-e725642fe1f3","Type":"ContainerDied","Data":"c5bf073cc763a1d9d28d71af483483518cb28ceae5a01d25323ff33cb76acb4a"} Jul 19 08:32:11.815744 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:11.815298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" event={"ID":"1f8d16c0-6637-40e0-80bf-e725642fe1f3","Type":"ContainerStarted","Data":"66c32edfdbc151e6465794807e2c6916ed7d1b7d757475f462f4e317cda1d9f5"} Jul 19 08:32:11.815744 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:11.815498 2574 scope.go:117] "RemoveContainer" containerID="c5bf073cc763a1d9d28d71af483483518cb28ceae5a01d25323ff33cb76acb4a" Jul 19 08:32:12.821785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:12.821699 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerID="14141c2e0142f681ea8c99d84a42c7c65625b956c3f7cfe97ae6c69c9ed8669e" exitCode=6 Jul 19 08:32:12.821785 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:12.821751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" event={"ID":"1f8d16c0-6637-40e0-80bf-e725642fe1f3","Type":"ContainerDied","Data":"14141c2e0142f681ea8c99d84a42c7c65625b956c3f7cfe97ae6c69c9ed8669e"} Jul 19 08:32:12.822240 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:12.821797 2574 scope.go:117] "RemoveContainer" containerID="c5bf073cc763a1d9d28d71af483483518cb28ceae5a01d25323ff33cb76acb4a" Jul 19 08:32:12.822240 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:12.821994 2574 scope.go:117] "RemoveContainer" containerID="14141c2e0142f681ea8c99d84a42c7c65625b956c3f7cfe97ae6c69c9ed8669e" Jul 19 08:32:12.822240 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:12.822199 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q_odh-ai-gateway-infra(1f8d16c0-6637-40e0-80bf-e725642fe1f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" Jul 19 08:32:13.828314 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:13.828286 2574 scope.go:117] "RemoveContainer" containerID="14141c2e0142f681ea8c99d84a42c7c65625b956c3f7cfe97ae6c69c9ed8669e" Jul 19 08:32:13.828747 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:13.828518 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q_odh-ai-gateway-infra(1f8d16c0-6637-40e0-80bf-e725642fe1f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" Jul 19 08:32:22.732570 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:22.732543 2574 scope.go:117] "RemoveContainer" containerID="e3336c62efd8b53dd56ad75e7191c4b89658ddb080489480a2aadef24c57df8b" Jul 19 08:32:23.876071 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:23.876036 2574 generic.go:356] "Generic (PLEG): container finished" podID="82096131-8ebe-49b7-85a2-80b545213919" containerID="e696a2a8a46eb3c945ca0b9e43ce88cc36e91a690c8f4be7beda3837ca55455c" exitCode=6 Jul 19 08:32:23.876518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:23.876103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" event={"ID":"82096131-8ebe-49b7-85a2-80b545213919","Type":"ContainerDied","Data":"e696a2a8a46eb3c945ca0b9e43ce88cc36e91a690c8f4be7beda3837ca55455c"} Jul 19 08:32:23.876518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:23.876167 2574 scope.go:117] "RemoveContainer" containerID="e3336c62efd8b53dd56ad75e7191c4b89658ddb080489480a2aadef24c57df8b" Jul 19 08:32:23.876518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:23.876458 2574 scope.go:117] "RemoveContainer" containerID="e696a2a8a46eb3c945ca0b9e43ce88cc36e91a690c8f4be7beda3837ca55455c" Jul 19 08:32:23.876785 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:23.876760 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2_odh-ai-gateway-infra(82096131-8ebe-49b7-85a2-80b545213919)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" podUID="82096131-8ebe-49b7-85a2-80b545213919" Jul 19 08:32:24.899825 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:24.899791 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2"] Jul 19 08:32:25.034858 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.034827 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:25.062277 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.062243 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82096131-8ebe-49b7-85a2-80b545213919-maas-api-service-ca\") pod \"82096131-8ebe-49b7-85a2-80b545213919\" (UID: \"82096131-8ebe-49b7-85a2-80b545213919\") " Jul 19 08:32:25.062636 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.062614 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82096131-8ebe-49b7-85a2-80b545213919-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82096131-8ebe-49b7-85a2-80b545213919" (UID: "82096131-8ebe-49b7-85a2-80b545213919"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:32:25.163953 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.163874 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82096131-8ebe-49b7-85a2-80b545213919-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:32:25.726851 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.726818 2574 scope.go:117] "RemoveContainer" containerID="14141c2e0142f681ea8c99d84a42c7c65625b956c3f7cfe97ae6c69c9ed8669e" Jul 19 08:32:25.890361 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.890336 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" Jul 19 08:32:25.890521 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.890381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2" event={"ID":"82096131-8ebe-49b7-85a2-80b545213919","Type":"ContainerDied","Data":"24afe140164ea2fc5c41f750a2a4739ff231afc7e94a036b9e2b32c8496c3b45"} Jul 19 08:32:25.890521 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.890446 2574 scope.go:117] "RemoveContainer" containerID="e696a2a8a46eb3c945ca0b9e43ce88cc36e91a690c8f4be7beda3837ca55455c" Jul 19 08:32:25.911389 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.911348 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2"] Jul 19 08:32:25.916748 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:25.916724 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-f66j2"] Jul 19 08:32:26.732047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:26.731961 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82096131-8ebe-49b7-85a2-80b545213919" path="/var/lib/kubelet/pods/82096131-8ebe-49b7-85a2-80b545213919/volumes" Jul 19 08:32:26.897411 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:26.897380 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerID="56b22a3196787ee6559b759c02c2c7ae349013c5ac9cc6eb3b663260f00fa3ac" exitCode=6 Jul 19 08:32:26.897594 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:26.897419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" event={"ID":"1f8d16c0-6637-40e0-80bf-e725642fe1f3","Type":"ContainerDied","Data":"56b22a3196787ee6559b759c02c2c7ae349013c5ac9cc6eb3b663260f00fa3ac"} Jul 19 08:32:26.897594 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:26.897467 2574 scope.go:117] "RemoveContainer" containerID="14141c2e0142f681ea8c99d84a42c7c65625b956c3f7cfe97ae6c69c9ed8669e" Jul 19 08:32:26.897759 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:26.897749 2574 scope.go:117] "RemoveContainer" containerID="56b22a3196787ee6559b759c02c2c7ae349013c5ac9cc6eb3b663260f00fa3ac" Jul 19 08:32:26.898030 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:26.898005 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q_odh-ai-gateway-infra(1f8d16c0-6637-40e0-80bf-e725642fe1f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" Jul 19 08:32:27.624450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624415 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2"] Jul 19 08:32:27.624867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624802 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.624867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624815 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.624867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624828 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.624867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624834 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.624867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624841 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.624867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624846 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.625081 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624904 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.625081 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.624911 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:32:27.627885 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.627868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:27.635197 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.634859 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2"] Jul 19 08:32:27.690250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.690211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db59696-2b79-4453-94b2-c4ef033c4204-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2\" (UID: \"5db59696-2b79-4453-94b2-c4ef033c4204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:27.791554 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.791522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db59696-2b79-4453-94b2-c4ef033c4204-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2\" (UID: \"5db59696-2b79-4453-94b2-c4ef033c4204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:27.792227 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.792208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db59696-2b79-4453-94b2-c4ef033c4204-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2\" (UID: \"5db59696-2b79-4453-94b2-c4ef033c4204\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:27.922687 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.922579 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q"] Jul 19 08:32:27.939452 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:27.939415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:28.081657 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.081628 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:28.109102 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.109068 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2"] Jul 19 08:32:28.111938 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:32:28.111910 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db59696_2b79_4453_94b2_c4ef033c4204.slice/crio-2b801f94eb1aaf44a7877f179d1d827d115f062da044abafb908194067e2be33 WatchSource:0}: Error finding container 2b801f94eb1aaf44a7877f179d1d827d115f062da044abafb908194067e2be33: Status 404 returned error can't find the container with id 2b801f94eb1aaf44a7877f179d1d827d115f062da044abafb908194067e2be33 Jul 19 08:32:28.194922 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.194824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8d16c0-6637-40e0-80bf-e725642fe1f3-maas-api-service-ca\") pod \"1f8d16c0-6637-40e0-80bf-e725642fe1f3\" (UID: \"1f8d16c0-6637-40e0-80bf-e725642fe1f3\") " Jul 19 08:32:28.195259 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.195226 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8d16c0-6637-40e0-80bf-e725642fe1f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f8d16c0-6637-40e0-80bf-e725642fe1f3" (UID: "1f8d16c0-6637-40e0-80bf-e725642fe1f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:32:28.295695 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.295644 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8d16c0-6637-40e0-80bf-e725642fe1f3-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:32:28.909582 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.909547 2574 generic.go:356] "Generic (PLEG): container finished" podID="5db59696-2b79-4453-94b2-c4ef033c4204" containerID="3e09ce45a560a8802240ca6f2b06a5c9a38c06f6c1767e440dd0ea48d0ec17d0" exitCode=6 Jul 19 08:32:28.910010 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.909636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" event={"ID":"5db59696-2b79-4453-94b2-c4ef033c4204","Type":"ContainerDied","Data":"3e09ce45a560a8802240ca6f2b06a5c9a38c06f6c1767e440dd0ea48d0ec17d0"} Jul 19 08:32:28.910010 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.909716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" event={"ID":"5db59696-2b79-4453-94b2-c4ef033c4204","Type":"ContainerStarted","Data":"2b801f94eb1aaf44a7877f179d1d827d115f062da044abafb908194067e2be33"} Jul 19 08:32:28.910010 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.909847 2574 scope.go:117] "RemoveContainer" containerID="3e09ce45a560a8802240ca6f2b06a5c9a38c06f6c1767e440dd0ea48d0ec17d0" Jul 19 08:32:28.911200 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.911176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" event={"ID":"1f8d16c0-6637-40e0-80bf-e725642fe1f3","Type":"ContainerDied","Data":"66c32edfdbc151e6465794807e2c6916ed7d1b7d757475f462f4e317cda1d9f5"} Jul 19 08:32:28.911200 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.911194 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q" Jul 19 08:32:28.911332 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.911236 2574 scope.go:117] "RemoveContainer" containerID="56b22a3196787ee6559b759c02c2c7ae349013c5ac9cc6eb3b663260f00fa3ac" Jul 19 08:32:28.935019 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.934992 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q"] Jul 19 08:32:28.938849 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:28.938826 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-l4h9q"] Jul 19 08:32:29.917620 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:29.917587 2574 generic.go:356] "Generic (PLEG): container finished" podID="5db59696-2b79-4453-94b2-c4ef033c4204" containerID="8746a214e7fec80d1d8344673c59a62ddb9cd42f115b87ccb3881206b5cbbf2a" exitCode=6 Jul 19 08:32:29.918112 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:29.917697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" event={"ID":"5db59696-2b79-4453-94b2-c4ef033c4204","Type":"ContainerDied","Data":"8746a214e7fec80d1d8344673c59a62ddb9cd42f115b87ccb3881206b5cbbf2a"} Jul 19 08:32:29.918112 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:29.917759 2574 scope.go:117] "RemoveContainer" containerID="3e09ce45a560a8802240ca6f2b06a5c9a38c06f6c1767e440dd0ea48d0ec17d0" Jul 19 08:32:29.918112 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:29.917962 2574 scope.go:117] "RemoveContainer" containerID="8746a214e7fec80d1d8344673c59a62ddb9cd42f115b87ccb3881206b5cbbf2a" Jul 19 08:32:29.918281 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:29.918202 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2_odh-ai-gateway-infra(5db59696-2b79-4453-94b2-c4ef033c4204)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" Jul 19 08:32:30.731915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:30.731835 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" path="/var/lib/kubelet/pods/1f8d16c0-6637-40e0-80bf-e725642fe1f3/volumes" Jul 19 08:32:30.925072 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:30.925042 2574 scope.go:117] "RemoveContainer" containerID="8746a214e7fec80d1d8344673c59a62ddb9cd42f115b87ccb3881206b5cbbf2a" Jul 19 08:32:30.925455 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:30.925224 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2_odh-ai-gateway-infra(5db59696-2b79-4453-94b2-c4ef033c4204)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" Jul 19 08:32:43.726259 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:43.726226 2574 scope.go:117] "RemoveContainer" containerID="8746a214e7fec80d1d8344673c59a62ddb9cd42f115b87ccb3881206b5cbbf2a" Jul 19 08:32:44.999574 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:44.999536 2574 generic.go:356] "Generic (PLEG): container finished" podID="5db59696-2b79-4453-94b2-c4ef033c4204" containerID="e5966d505109a42e9c74f52ef5de85db33551d1c5cfcb4395da61182aa769377" exitCode=6 Jul 19 08:32:45.000039 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:44.999609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" event={"ID":"5db59696-2b79-4453-94b2-c4ef033c4204","Type":"ContainerDied","Data":"e5966d505109a42e9c74f52ef5de85db33551d1c5cfcb4395da61182aa769377"} Jul 19 08:32:45.000039 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:44.999720 2574 scope.go:117] "RemoveContainer" containerID="8746a214e7fec80d1d8344673c59a62ddb9cd42f115b87ccb3881206b5cbbf2a" Jul 19 08:32:45.000152 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:45.000062 2574 scope.go:117] "RemoveContainer" containerID="e5966d505109a42e9c74f52ef5de85db33551d1c5cfcb4395da61182aa769377" Jul 19 08:32:45.000325 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:32:45.000299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2_odh-ai-gateway-infra(5db59696-2b79-4453-94b2-c4ef033c4204)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" Jul 19 08:32:46.022665 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:46.022625 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2"] Jul 19 08:32:46.159186 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:46.159161 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:46.267799 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:46.267767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db59696-2b79-4453-94b2-c4ef033c4204-maas-api-service-ca\") pod \"5db59696-2b79-4453-94b2-c4ef033c4204\" (UID: \"5db59696-2b79-4453-94b2-c4ef033c4204\") " Jul 19 08:32:46.268149 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:46.268127 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db59696-2b79-4453-94b2-c4ef033c4204-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5db59696-2b79-4453-94b2-c4ef033c4204" (UID: "5db59696-2b79-4453-94b2-c4ef033c4204"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:32:46.369316 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:46.369279 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db59696-2b79-4453-94b2-c4ef033c4204-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:32:47.012340 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:47.012259 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" Jul 19 08:32:47.012340 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:47.012301 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2" event={"ID":"5db59696-2b79-4453-94b2-c4ef033c4204","Type":"ContainerDied","Data":"2b801f94eb1aaf44a7877f179d1d827d115f062da044abafb908194067e2be33"} Jul 19 08:32:47.012529 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:47.012365 2574 scope.go:117] "RemoveContainer" containerID="e5966d505109a42e9c74f52ef5de85db33551d1c5cfcb4395da61182aa769377" Jul 19 08:32:47.029601 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:47.029575 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2"] Jul 19 08:32:47.034539 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:47.034514 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-k2hz2"] Jul 19 08:32:48.733506 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:32:48.733469 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" path="/var/lib/kubelet/pods/5db59696-2b79-4453-94b2-c4ef033c4204/volumes" Jul 19 08:33:00.396426 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396387 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz"] Jul 19 08:33:00.396835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396791 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.396835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396804 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.396835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396814 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.396835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396820 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.396835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396828 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.396835 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396833 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396849 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396855 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396865 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396870 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396934 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="82096131-8ebe-49b7-85a2-80b545213919" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396942 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396948 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396954 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396959 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.397029 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.396966 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:00.401272 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.401255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:00.403958 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.403929 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:33:00.405962 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.405939 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz"] Jul 19 08:33:00.493878 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.493840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0c149-bc53-4ff5-906d-a63bcc3e88b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz\" (UID: \"48e0c149-bc53-4ff5-906d-a63bcc3e88b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:00.594714 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.594649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0c149-bc53-4ff5-906d-a63bcc3e88b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz\" (UID: \"48e0c149-bc53-4ff5-906d-a63bcc3e88b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:00.595295 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.595274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0c149-bc53-4ff5-906d-a63bcc3e88b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz\" (UID: \"48e0c149-bc53-4ff5-906d-a63bcc3e88b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:00.713829 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.713740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:00.847272 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:00.847058 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz"] Jul 19 08:33:00.849600 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:33:00.849575 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e0c149_bc53_4ff5_906d_a63bcc3e88b3.slice/crio-781c9d0f101cf5554c4b3df6136a156f21faf5e311360a7a6cf6d005857de903 WatchSource:0}: Error finding container 781c9d0f101cf5554c4b3df6136a156f21faf5e311360a7a6cf6d005857de903: Status 404 returned error can't find the container with id 781c9d0f101cf5554c4b3df6136a156f21faf5e311360a7a6cf6d005857de903 Jul 19 08:33:01.078778 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:01.078663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" event={"ID":"48e0c149-bc53-4ff5-906d-a63bcc3e88b3","Type":"ContainerStarted","Data":"781c9d0f101cf5554c4b3df6136a156f21faf5e311360a7a6cf6d005857de903"} Jul 19 08:33:02.084300 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:02.084261 2574 generic.go:356] "Generic (PLEG): container finished" podID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerID="49edbbce9400c6883c77f5da3dd9e2dee8b3be5a2c384eb174ab9ffb4bd2d45e" exitCode=6 Jul 19 08:33:02.084705 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:02.084309 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" event={"ID":"48e0c149-bc53-4ff5-906d-a63bcc3e88b3","Type":"ContainerDied","Data":"49edbbce9400c6883c77f5da3dd9e2dee8b3be5a2c384eb174ab9ffb4bd2d45e"} Jul 19 08:33:02.084705 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:02.084573 2574 scope.go:117] "RemoveContainer" containerID="49edbbce9400c6883c77f5da3dd9e2dee8b3be5a2c384eb174ab9ffb4bd2d45e" Jul 19 08:33:03.091130 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:03.091095 2574 generic.go:356] "Generic (PLEG): container finished" podID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerID="b0207682e461f643cabcffdb9a283221c0fd57277dde8c0d9822acce92deed75" exitCode=6 Jul 19 08:33:03.091559 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:03.091185 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" event={"ID":"48e0c149-bc53-4ff5-906d-a63bcc3e88b3","Type":"ContainerDied","Data":"b0207682e461f643cabcffdb9a283221c0fd57277dde8c0d9822acce92deed75"} Jul 19 08:33:03.091559 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:03.091241 2574 scope.go:117] "RemoveContainer" containerID="49edbbce9400c6883c77f5da3dd9e2dee8b3be5a2c384eb174ab9ffb4bd2d45e" Jul 19 08:33:03.091559 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:03.091404 2574 scope.go:117] "RemoveContainer" containerID="b0207682e461f643cabcffdb9a283221c0fd57277dde8c0d9822acce92deed75" Jul 19 08:33:03.091743 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:03.091627 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz_odh-ai-gateway-infra(48e0c149-bc53-4ff5-906d-a63bcc3e88b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" Jul 19 08:33:04.097537 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.097511 2574 scope.go:117] "RemoveContainer" containerID="b0207682e461f643cabcffdb9a283221c0fd57277dde8c0d9822acce92deed75" Jul 19 08:33:04.097944 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:04.097766 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz_odh-ai-gateway-infra(48e0c149-bc53-4ff5-906d-a63bcc3e88b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" Jul 19 08:33:04.571215 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.571122 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx"] Jul 19 08:33:04.571530 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.571517 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:04.571530 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.571531 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8d16c0-6637-40e0-80bf-e725642fe1f3" containerName="revoke-keys" Jul 19 08:33:04.571665 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.571606 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db59696-2b79-4453-94b2-c4ef033c4204" containerName="revoke-keys" Jul 19 08:33:04.575953 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.575926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:04.581836 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.581806 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx"] Jul 19 08:33:04.633243 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.633212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx\" (UID: \"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:04.734469 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.734425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx\" (UID: \"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:04.735076 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.735056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx\" (UID: \"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:04.889006 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:04.888959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:05.021753 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.021729 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx"] Jul 19 08:33:05.023665 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:33:05.023639 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b32af2f_9bd2_4e4d_8905_d2fa8d23e0c2.slice/crio-c04ae2616bef55b61fdc27c5a6a2c7d50cf47dc3342fdd43718813e9d29e49d9 WatchSource:0}: Error finding container c04ae2616bef55b61fdc27c5a6a2c7d50cf47dc3342fdd43718813e9d29e49d9: Status 404 returned error can't find the container with id c04ae2616bef55b61fdc27c5a6a2c7d50cf47dc3342fdd43718813e9d29e49d9 Jul 19 08:33:05.104063 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.104020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" event={"ID":"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2","Type":"ContainerStarted","Data":"c04ae2616bef55b61fdc27c5a6a2c7d50cf47dc3342fdd43718813e9d29e49d9"} Jul 19 08:33:05.327523 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.327484 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2"] Jul 19 08:33:05.331805 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.331780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:05.335278 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.335249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2"] Jul 19 08:33:05.440823 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.440783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f61d5-5ed0-4936-9e30-b5f59b8142b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2\" (UID: \"a51f61d5-5ed0-4936-9e30-b5f59b8142b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:05.542227 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.542122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f61d5-5ed0-4936-9e30-b5f59b8142b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2\" (UID: \"a51f61d5-5ed0-4936-9e30-b5f59b8142b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:05.542880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.542856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f61d5-5ed0-4936-9e30-b5f59b8142b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2\" (UID: \"a51f61d5-5ed0-4936-9e30-b5f59b8142b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:05.643938 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.643909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:05.780041 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:05.780004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2"] Jul 19 08:33:05.781473 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:33:05.781449 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51f61d5_5ed0_4936_9e30_b5f59b8142b3.slice/crio-36e4c20d4fcb2f0e7e5f7f84ec18715cd1b0ae46bf3b5902e60c62304ce1aff6 WatchSource:0}: Error finding container 36e4c20d4fcb2f0e7e5f7f84ec18715cd1b0ae46bf3b5902e60c62304ce1aff6: Status 404 returned error can't find the container with id 36e4c20d4fcb2f0e7e5f7f84ec18715cd1b0ae46bf3b5902e60c62304ce1aff6 Jul 19 08:33:06.111312 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:06.111274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerStarted","Data":"8598953e73d903576340bb020f65034054247c3ccffaff854db53478287642e2"} Jul 19 08:33:06.111849 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:06.111333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerStarted","Data":"36e4c20d4fcb2f0e7e5f7f84ec18715cd1b0ae46bf3b5902e60c62304ce1aff6"} Jul 19 08:33:06.112828 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:06.112802 2574 generic.go:356] "Generic (PLEG): container finished" podID="8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2" containerID="bb21e02540a5ccea07810379fdda67ae3a0d1b56d374ea00470e4c7988855b0c" exitCode=0 Jul 19 08:33:06.112933 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:06.112850 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" event={"ID":"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2","Type":"ContainerDied","Data":"bb21e02540a5ccea07810379fdda67ae3a0d1b56d374ea00470e4c7988855b0c"} Jul 19 08:33:06.124894 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:06.124822 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" podStartSLOduration=1.124804203 podStartE2EDuration="1.124804203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:06.123932196 +0000 UTC m=+3033.903595510" watchObservedRunningTime="2026-07-19 08:33:06.124804203 +0000 UTC m=+3033.904467597" Jul 19 08:33:07.119698 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.119653 2574 generic.go:356] "Generic (PLEG): container finished" podID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerID="8598953e73d903576340bb020f65034054247c3ccffaff854db53478287642e2" exitCode=6 Jul 19 08:33:07.120079 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.119744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerDied","Data":"8598953e73d903576340bb020f65034054247c3ccffaff854db53478287642e2"} Jul 19 08:33:07.120079 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.119953 2574 scope.go:117] "RemoveContainer" containerID="8598953e73d903576340bb020f65034054247c3ccffaff854db53478287642e2" Jul 19 08:33:07.268312 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.268288 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:07.358842 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.358806 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2-maas-api-service-ca\") pod \"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2\" (UID: \"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2\") " Jul 19 08:33:07.359203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.359177 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2" (UID: "8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:07.459728 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:07.459610 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:33:08.126454 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.126414 2574 generic.go:356] "Generic (PLEG): container finished" podID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerID="0fbbd459669aa98761d948507054503950e0d396f4a24da615137e8966ac62d5" exitCode=6 Jul 19 08:33:08.126957 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.126501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerDied","Data":"0fbbd459669aa98761d948507054503950e0d396f4a24da615137e8966ac62d5"} Jul 19 08:33:08.126957 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.126563 2574 scope.go:117] "RemoveContainer" containerID="8598953e73d903576340bb020f65034054247c3ccffaff854db53478287642e2" Jul 19 08:33:08.126957 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.126767 2574 scope.go:117] "RemoveContainer" containerID="0fbbd459669aa98761d948507054503950e0d396f4a24da615137e8966ac62d5" Jul 19 08:33:08.127135 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:08.126998 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2_odh-ai-gateway-infra(a51f61d5-5ed0-4936-9e30-b5f59b8142b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" Jul 19 08:33:08.127843 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.127825 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" Jul 19 08:33:08.127926 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.127826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx" event={"ID":"8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2","Type":"ContainerDied","Data":"c04ae2616bef55b61fdc27c5a6a2c7d50cf47dc3342fdd43718813e9d29e49d9"} Jul 19 08:33:08.127978 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:08.127947 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04ae2616bef55b61fdc27c5a6a2c7d50cf47dc3342fdd43718813e9d29e49d9" Jul 19 08:33:09.133894 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:09.133869 2574 scope.go:117] "RemoveContainer" containerID="0fbbd459669aa98761d948507054503950e0d396f4a24da615137e8966ac62d5" Jul 19 08:33:09.134270 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:09.134057 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2_odh-ai-gateway-infra(a51f61d5-5ed0-4936-9e30-b5f59b8142b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" Jul 19 08:33:15.726797 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:15.726758 2574 scope.go:117] "RemoveContainer" containerID="b0207682e461f643cabcffdb9a283221c0fd57277dde8c0d9822acce92deed75" Jul 19 08:33:16.168667 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:16.168632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" event={"ID":"48e0c149-bc53-4ff5-906d-a63bcc3e88b3","Type":"ContainerStarted","Data":"3bb2ca5114fb237822058f422ca18b26f55309298b1250cce75cc70c56df8983"} Jul 19 08:33:16.184732 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:16.184659 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" podStartSLOduration=16.184641659 podStartE2EDuration="16.184641659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:16.182159204 +0000 UTC m=+3043.961822521" watchObservedRunningTime="2026-07-19 08:33:16.184641659 +0000 UTC m=+3043.964304972" Jul 19 08:33:17.174505 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.174470 2574 generic.go:356] "Generic (PLEG): container finished" podID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerID="3bb2ca5114fb237822058f422ca18b26f55309298b1250cce75cc70c56df8983" exitCode=6 Jul 19 08:33:17.174957 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.174546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" event={"ID":"48e0c149-bc53-4ff5-906d-a63bcc3e88b3","Type":"ContainerDied","Data":"3bb2ca5114fb237822058f422ca18b26f55309298b1250cce75cc70c56df8983"} Jul 19 08:33:17.174957 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.174609 2574 scope.go:117] "RemoveContainer" containerID="b0207682e461f643cabcffdb9a283221c0fd57277dde8c0d9822acce92deed75" Jul 19 08:33:17.175092 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.174984 2574 scope.go:117] "RemoveContainer" containerID="3bb2ca5114fb237822058f422ca18b26f55309298b1250cce75cc70c56df8983" Jul 19 08:33:17.175282 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:17.175250 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz_odh-ai-gateway-infra(48e0c149-bc53-4ff5-906d-a63bcc3e88b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" Jul 19 08:33:17.197803 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.197775 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz"] Jul 19 08:33:17.673060 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.673025 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl"] Jul 19 08:33:17.673421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.673409 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2" containerName="revoke-keys" Jul 19 08:33:17.673473 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.673422 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2" containerName="revoke-keys" Jul 19 08:33:17.673516 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.673485 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2" containerName="revoke-keys" Jul 19 08:33:17.677748 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.677727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:17.683561 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.683533 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl"] Jul 19 08:33:17.757112 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.757070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd064363-644b-4d4e-a87b-4974710320a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-lncrl\" (UID: \"bd064363-644b-4d4e-a87b-4974710320a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:17.858470 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.858419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd064363-644b-4d4e-a87b-4974710320a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-lncrl\" (UID: \"bd064363-644b-4d4e-a87b-4974710320a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:17.859151 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.859126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd064363-644b-4d4e-a87b-4974710320a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-lncrl\" (UID: \"bd064363-644b-4d4e-a87b-4974710320a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:17.991435 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:17.991337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:18.125015 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:18.124990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl"] Jul 19 08:33:18.127364 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:33:18.127340 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd064363_644b_4d4e_a87b_4974710320a0.slice/crio-bee83f3f9ff09fc3468219a90730239874ebb17b390db77cae660a189ed8adc7 WatchSource:0}: Error finding container bee83f3f9ff09fc3468219a90730239874ebb17b390db77cae660a189ed8adc7: Status 404 returned error can't find the container with id bee83f3f9ff09fc3468219a90730239874ebb17b390db77cae660a189ed8adc7 Jul 19 08:33:18.183032 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:18.183005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" event={"ID":"bd064363-644b-4d4e-a87b-4974710320a0","Type":"ContainerStarted","Data":"bee83f3f9ff09fc3468219a90730239874ebb17b390db77cae660a189ed8adc7"} Jul 19 08:33:18.313490 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:18.313463 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:18.464852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:18.464806 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0c149-bc53-4ff5-906d-a63bcc3e88b3-maas-api-service-ca\") pod \"48e0c149-bc53-4ff5-906d-a63bcc3e88b3\" (UID: \"48e0c149-bc53-4ff5-906d-a63bcc3e88b3\") " Jul 19 08:33:18.465232 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:18.465207 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e0c149-bc53-4ff5-906d-a63bcc3e88b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48e0c149-bc53-4ff5-906d-a63bcc3e88b3" (UID: "48e0c149-bc53-4ff5-906d-a63bcc3e88b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:18.566568 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:18.566477 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0c149-bc53-4ff5-906d-a63bcc3e88b3-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:33:19.189382 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.189326 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" Jul 19 08:33:19.189904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.189370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz" event={"ID":"48e0c149-bc53-4ff5-906d-a63bcc3e88b3","Type":"ContainerDied","Data":"781c9d0f101cf5554c4b3df6136a156f21faf5e311360a7a6cf6d005857de903"} Jul 19 08:33:19.189904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.189473 2574 scope.go:117] "RemoveContainer" containerID="3bb2ca5114fb237822058f422ca18b26f55309298b1250cce75cc70c56df8983" Jul 19 08:33:19.191058 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.191035 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd064363-644b-4d4e-a87b-4974710320a0" containerID="5cf87450e3f81f6a3bd1b42220faa1bb1ac2dd0004ef8020953a7a3be924dbe1" exitCode=6 Jul 19 08:33:19.191174 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.191091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" event={"ID":"bd064363-644b-4d4e-a87b-4974710320a0","Type":"ContainerDied","Data":"5cf87450e3f81f6a3bd1b42220faa1bb1ac2dd0004ef8020953a7a3be924dbe1"} Jul 19 08:33:19.191468 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.191445 2574 scope.go:117] "RemoveContainer" containerID="5cf87450e3f81f6a3bd1b42220faa1bb1ac2dd0004ef8020953a7a3be924dbe1" Jul 19 08:33:19.220838 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.220809 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz"] Jul 19 08:33:19.223613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:19.223588 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hqlbz"] Jul 19 08:33:20.197858 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:20.197822 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd064363-644b-4d4e-a87b-4974710320a0" containerID="90a73fa1174ef0559a5163bfc153ec588b6d852fa1e4ce18ad4097490482c849" exitCode=6 Jul 19 08:33:20.198328 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:20.197867 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" event={"ID":"bd064363-644b-4d4e-a87b-4974710320a0","Type":"ContainerDied","Data":"90a73fa1174ef0559a5163bfc153ec588b6d852fa1e4ce18ad4097490482c849"} Jul 19 08:33:20.198328 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:20.197926 2574 scope.go:117] "RemoveContainer" containerID="5cf87450e3f81f6a3bd1b42220faa1bb1ac2dd0004ef8020953a7a3be924dbe1" Jul 19 08:33:20.198328 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:20.198116 2574 scope.go:117] "RemoveContainer" containerID="90a73fa1174ef0559a5163bfc153ec588b6d852fa1e4ce18ad4097490482c849" Jul 19 08:33:20.198446 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:20.198332 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-lncrl_odh-ai-gateway-infra(bd064363-644b-4d4e-a87b-4974710320a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" podUID="bd064363-644b-4d4e-a87b-4974710320a0" Jul 19 08:33:20.732322 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:20.732287 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" path="/var/lib/kubelet/pods/48e0c149-bc53-4ff5-906d-a63bcc3e88b3/volumes" Jul 19 08:33:21.204855 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:21.204825 2574 scope.go:117] "RemoveContainer" containerID="90a73fa1174ef0559a5163bfc153ec588b6d852fa1e4ce18ad4097490482c849" Jul 19 08:33:21.205232 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:21.205036 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-lncrl_odh-ai-gateway-infra(bd064363-644b-4d4e-a87b-4974710320a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" podUID="bd064363-644b-4d4e-a87b-4974710320a0" Jul 19 08:33:23.726475 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:23.726392 2574 scope.go:117] "RemoveContainer" containerID="0fbbd459669aa98761d948507054503950e0d396f4a24da615137e8966ac62d5" Jul 19 08:33:24.221706 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:24.221655 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerStarted","Data":"5b261dde8274d6532ed7c283fd2eca2f6b0a73a11f52cb32ec063321b9dcef95"} Jul 19 08:33:25.228393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:25.228349 2574 generic.go:356] "Generic (PLEG): container finished" podID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerID="5b261dde8274d6532ed7c283fd2eca2f6b0a73a11f52cb32ec063321b9dcef95" exitCode=6 Jul 19 08:33:25.228816 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:25.228426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerDied","Data":"5b261dde8274d6532ed7c283fd2eca2f6b0a73a11f52cb32ec063321b9dcef95"} Jul 19 08:33:25.228816 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:25.228513 2574 scope.go:117] "RemoveContainer" containerID="0fbbd459669aa98761d948507054503950e0d396f4a24da615137e8966ac62d5" Jul 19 08:33:25.228923 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:25.228823 2574 scope.go:117] "RemoveContainer" containerID="5b261dde8274d6532ed7c283fd2eca2f6b0a73a11f52cb32ec063321b9dcef95" Jul 19 08:33:25.229079 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:25.229057 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2_odh-ai-gateway-infra(a51f61d5-5ed0-4936-9e30-b5f59b8142b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" Jul 19 08:33:25.244963 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:25.244937 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2"] Jul 19 08:33:26.362950 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:26.362922 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:26.441963 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:26.441930 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f61d5-5ed0-4936-9e30-b5f59b8142b3-maas-api-service-ca\") pod \"a51f61d5-5ed0-4936-9e30-b5f59b8142b3\" (UID: \"a51f61d5-5ed0-4936-9e30-b5f59b8142b3\") " Jul 19 08:33:26.442356 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:26.442333 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51f61d5-5ed0-4936-9e30-b5f59b8142b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a51f61d5-5ed0-4936-9e30-b5f59b8142b3" (UID: "a51f61d5-5ed0-4936-9e30-b5f59b8142b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:26.543234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:26.543149 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51f61d5-5ed0-4936-9e30-b5f59b8142b3-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:33:27.240316 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:27.240283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" event={"ID":"a51f61d5-5ed0-4936-9e30-b5f59b8142b3","Type":"ContainerDied","Data":"36e4c20d4fcb2f0e7e5f7f84ec18715cd1b0ae46bf3b5902e60c62304ce1aff6"} Jul 19 08:33:27.240499 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:27.240325 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2" Jul 19 08:33:27.240499 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:27.240344 2574 scope.go:117] "RemoveContainer" containerID="5b261dde8274d6532ed7c283fd2eca2f6b0a73a11f52cb32ec063321b9dcef95" Jul 19 08:33:27.256051 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:27.256023 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2"] Jul 19 08:33:27.261711 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:27.261672 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-qkbg2"] Jul 19 08:33:28.732300 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:28.732263 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" path="/var/lib/kubelet/pods/a51f61d5-5ed0-4936-9e30-b5f59b8142b3/volumes" Jul 19 08:33:35.726993 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:35.726962 2574 scope.go:117] "RemoveContainer" containerID="90a73fa1174ef0559a5163bfc153ec588b6d852fa1e4ce18ad4097490482c849" Jul 19 08:33:36.292176 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:36.292138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" event={"ID":"bd064363-644b-4d4e-a87b-4974710320a0","Type":"ContainerStarted","Data":"11165cd4429b13eb34b43312bfe8c30eb559d044f69afd6576b36dfb5e6503b6"} Jul 19 08:33:36.309216 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:36.309166 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" podStartSLOduration=19.30915006 podStartE2EDuration="19.30915006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:36.307229491 +0000 UTC m=+3064.086892830" watchObservedRunningTime="2026-07-19 08:33:36.30915006 +0000 UTC m=+3064.088813415" Jul 19 08:33:37.298488 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:37.298450 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd064363-644b-4d4e-a87b-4974710320a0" containerID="11165cd4429b13eb34b43312bfe8c30eb559d044f69afd6576b36dfb5e6503b6" exitCode=6 Jul 19 08:33:37.298949 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:37.298524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" event={"ID":"bd064363-644b-4d4e-a87b-4974710320a0","Type":"ContainerDied","Data":"11165cd4429b13eb34b43312bfe8c30eb559d044f69afd6576b36dfb5e6503b6"} Jul 19 08:33:37.298949 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:37.298589 2574 scope.go:117] "RemoveContainer" containerID="90a73fa1174ef0559a5163bfc153ec588b6d852fa1e4ce18ad4097490482c849" Jul 19 08:33:37.299046 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:37.298974 2574 scope.go:117] "RemoveContainer" containerID="11165cd4429b13eb34b43312bfe8c30eb559d044f69afd6576b36dfb5e6503b6" Jul 19 08:33:37.299268 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:37.299242 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-lncrl_odh-ai-gateway-infra(bd064363-644b-4d4e-a87b-4974710320a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" podUID="bd064363-644b-4d4e-a87b-4974710320a0" Jul 19 08:33:37.323865 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:37.323827 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl"] Jul 19 08:33:38.437517 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:38.437486 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:38.551889 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:38.551850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd064363-644b-4d4e-a87b-4974710320a0-maas-api-service-ca\") pod \"bd064363-644b-4d4e-a87b-4974710320a0\" (UID: \"bd064363-644b-4d4e-a87b-4974710320a0\") " Jul 19 08:33:38.552219 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:38.552194 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd064363-644b-4d4e-a87b-4974710320a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd064363-644b-4d4e-a87b-4974710320a0" (UID: "bd064363-644b-4d4e-a87b-4974710320a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:33:38.653002 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:38.652960 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd064363-644b-4d4e-a87b-4974710320a0-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:33:39.311808 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:39.311773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" event={"ID":"bd064363-644b-4d4e-a87b-4974710320a0","Type":"ContainerDied","Data":"bee83f3f9ff09fc3468219a90730239874ebb17b390db77cae660a189ed8adc7"} Jul 19 08:33:39.311808 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:39.311807 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl" Jul 19 08:33:39.312087 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:39.311834 2574 scope.go:117] "RemoveContainer" containerID="11165cd4429b13eb34b43312bfe8c30eb559d044f69afd6576b36dfb5e6503b6" Jul 19 08:33:39.328082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:39.328055 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl"] Jul 19 08:33:39.334557 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:39.334530 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-lncrl"] Jul 19 08:33:40.731418 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:40.731381 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd064363-644b-4d4e-a87b-4974710320a0" path="/var/lib/kubelet/pods/bd064363-644b-4d4e-a87b-4974710320a0/volumes" Jul 19 08:33:42.603191 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603152 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8"] Jul 19 08:33:42.603716 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603665 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603719 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603736 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603746 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603763 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603771 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603783 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603791 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603801 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603809 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.603819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603817 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603825 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603837 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603845 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603875 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.603882 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604001 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604012 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604020 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604029 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604040 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604049 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.604383 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.604057 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:42.607278 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.607256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.609855 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.609831 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-190fde-serving-cert\"" Jul 19 08:33:42.617639 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.617614 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8"] Jul 19 08:33:42.690813 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.690780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbt2n\" (UniqueName: \"kubernetes.io/projected/7312a3cc-99e7-44de-80d9-3c7b046b3727-kube-api-access-fbt2n\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.691002 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.690840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312a3cc-99e7-44de-80d9-3c7b046b3727-openshift-service-ca\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.691002 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.690884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.792102 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.792058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt2n\" (UniqueName: \"kubernetes.io/projected/7312a3cc-99e7-44de-80d9-3c7b046b3727-kube-api-access-fbt2n\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.792278 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.792135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312a3cc-99e7-44de-80d9-3c7b046b3727-openshift-service-ca\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.792278 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.792166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.792351 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:42.792280 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-serving-cert: secret "maas-api-e2e-shared-b-190fde-serving-cert" not found Jul 19 08:33:42.792396 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:42.792385 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls podName:7312a3cc-99e7-44de-80d9-3c7b046b3727 nodeName:}" failed. No retries permitted until 2026-07-19 08:33:43.292325954 +0000 UTC m=+3071.071989253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls") pod "maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" (UID: "7312a3cc-99e7-44de-80d9-3c7b046b3727") : secret "maas-api-e2e-shared-b-190fde-serving-cert" not found Jul 19 08:33:42.792897 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.792874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312a3cc-99e7-44de-80d9-3c7b046b3727-openshift-service-ca\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:42.814364 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:42.814341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt2n\" (UniqueName: \"kubernetes.io/projected/7312a3cc-99e7-44de-80d9-3c7b046b3727-kube-api-access-fbt2n\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:43.297015 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:43.296976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:43.299863 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:43.299837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls\") pod \"maas-api-e2e-shared-b-190fde-786d85c676-wfrv8\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:43.518028 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:43.517990 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:43.656914 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:43.656889 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8"] Jul 19 08:33:43.658870 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:33:43.658836 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7312a3cc_99e7_44de_80d9_3c7b046b3727.slice/crio-13696b56d501a94d1b5664b626707296480f88bba12bf2f3700fb6bfb964f73d WatchSource:0}: Error finding container 13696b56d501a94d1b5664b626707296480f88bba12bf2f3700fb6bfb964f73d: Status 404 returned error can't find the container with id 13696b56d501a94d1b5664b626707296480f88bba12bf2f3700fb6bfb964f73d Jul 19 08:33:43.660450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:43.660430 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 08:33:44.343085 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:44.343052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" event={"ID":"7312a3cc-99e7-44de-80d9-3c7b046b3727","Type":"ContainerStarted","Data":"694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7"} Jul 19 08:33:44.343271 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:44.343103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" event={"ID":"7312a3cc-99e7-44de-80d9-3c7b046b3727","Type":"ContainerStarted","Data":"13696b56d501a94d1b5664b626707296480f88bba12bf2f3700fb6bfb964f73d"} Jul 19 08:33:44.343271 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:44.343159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:44.359573 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:44.359509 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" podStartSLOduration=2.027126753 podStartE2EDuration="2.359495315s" totalImagesPullingTime="332.368562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:33:44.358746533 +0000 UTC m=+3072.138409873" watchObservedRunningTime="2026-07-19 08:33:44.359495315 +0000 UTC m=+3072.139158629" Jul 19 08:33:50.353748 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:50.353715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:33:53.682191 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.682157 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh"] Jul 19 08:33:53.682591 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.682528 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:53.682591 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.682538 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e0c149-bc53-4ff5-906d-a63bcc3e88b3" containerName="revoke-keys" Jul 19 08:33:53.682664 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.682607 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51f61d5-5ed0-4936-9e30-b5f59b8142b3" containerName="revoke-keys" Jul 19 08:33:53.682664 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.682616 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd064363-644b-4d4e-a87b-4974710320a0" containerName="revoke-keys" Jul 19 08:33:53.685847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.685831 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:33:53.687983 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.687953 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:33:53.691583 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.691558 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh"] Jul 19 08:33:53.801116 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.801073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b02bd-e294-47a7-8262-eeb1d906c25d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh\" (UID: \"2a4b02bd-e294-47a7-8262-eeb1d906c25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:33:53.902114 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.902075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b02bd-e294-47a7-8262-eeb1d906c25d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh\" (UID: \"2a4b02bd-e294-47a7-8262-eeb1d906c25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:33:53.902793 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.902774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b02bd-e294-47a7-8262-eeb1d906c25d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh\" (UID: \"2a4b02bd-e294-47a7-8262-eeb1d906c25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:33:53.997026 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:53.996940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:33:54.131096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:54.131067 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh"] Jul 19 08:33:54.132858 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:33:54.132829 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b02bd_e294_47a7_8262_eeb1d906c25d.slice/crio-549d88ed7275c106d5d721637bd315408e6fc96d58c9024db143eebf6ec3f504 WatchSource:0}: Error finding container 549d88ed7275c106d5d721637bd315408e6fc96d58c9024db143eebf6ec3f504: Status 404 returned error can't find the container with id 549d88ed7275c106d5d721637bd315408e6fc96d58c9024db143eebf6ec3f504 Jul 19 08:33:54.392600 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:54.392563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" event={"ID":"2a4b02bd-e294-47a7-8262-eeb1d906c25d","Type":"ContainerStarted","Data":"549d88ed7275c106d5d721637bd315408e6fc96d58c9024db143eebf6ec3f504"} Jul 19 08:33:55.399096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:55.399059 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerID="a504fef3f2e240664b5ef75fb03014340fba611b6c89066325553bfc2496656c" exitCode=6 Jul 19 08:33:55.399096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:55.399097 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" event={"ID":"2a4b02bd-e294-47a7-8262-eeb1d906c25d","Type":"ContainerDied","Data":"a504fef3f2e240664b5ef75fb03014340fba611b6c89066325553bfc2496656c"} Jul 19 08:33:55.399571 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:55.399310 2574 scope.go:117] "RemoveContainer" containerID="a504fef3f2e240664b5ef75fb03014340fba611b6c89066325553bfc2496656c" Jul 19 08:33:56.405317 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:56.405277 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerID="1e0b2dfe6e62f97bc842f76b18aa3b45e3ae8d62d7184ef428ef08695dd4754c" exitCode=6 Jul 19 08:33:56.405774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:56.405363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" event={"ID":"2a4b02bd-e294-47a7-8262-eeb1d906c25d","Type":"ContainerDied","Data":"1e0b2dfe6e62f97bc842f76b18aa3b45e3ae8d62d7184ef428ef08695dd4754c"} Jul 19 08:33:56.405774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:56.405460 2574 scope.go:117] "RemoveContainer" containerID="a504fef3f2e240664b5ef75fb03014340fba611b6c89066325553bfc2496656c" Jul 19 08:33:56.405774 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:56.405707 2574 scope.go:117] "RemoveContainer" containerID="1e0b2dfe6e62f97bc842f76b18aa3b45e3ae8d62d7184ef428ef08695dd4754c" Jul 19 08:33:56.405987 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:56.405965 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh_odh-ai-gateway-infra(2a4b02bd-e294-47a7-8262-eeb1d906c25d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" Jul 19 08:33:57.411964 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:33:57.411935 2574 scope.go:117] "RemoveContainer" containerID="1e0b2dfe6e62f97bc842f76b18aa3b45e3ae8d62d7184ef428ef08695dd4754c" Jul 19 08:33:57.412349 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:33:57.412163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh_odh-ai-gateway-infra(2a4b02bd-e294-47a7-8262-eeb1d906c25d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" Jul 19 08:34:06.163904 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.163870 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc"] Jul 19 08:34:06.168781 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.168748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.171979 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.171957 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72ce60\"/\"default-dockercfg-vv5qw\"" Jul 19 08:34:06.172117 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.171955 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-72ce60\"/\"openshift-service-ca.crt\"" Jul 19 08:34:06.172117 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.171956 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-72ce60\"/\"kube-root-ca.crt\"" Jul 19 08:34:06.172117 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.171962 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72ce60\"/\"auth-test-model-72ce60-kserve-self-signed-certs\"" Jul 19 08:34:06.176122 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.176095 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc"] Jul 19 08:34:06.323009 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.322971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-model-cache\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.323009 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.323014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-home\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.323282 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.323039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea738c1-cce2-48f2-be41-fd25b306ab54-tls-certs\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.323282 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.323165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-dshm\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.323282 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.323201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wmq\" (UniqueName: \"kubernetes.io/projected/2ea738c1-cce2-48f2-be41-fd25b306ab54-kube-api-access-44wmq\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.323282 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.323230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-tmp-dir\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424135 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-home\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424135 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea738c1-cce2-48f2-be41-fd25b306ab54-tls-certs\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424370 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-dshm\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424370 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wmq\" (UniqueName: \"kubernetes.io/projected/2ea738c1-cce2-48f2-be41-fd25b306ab54-kube-api-access-44wmq\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424370 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-tmp-dir\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424543 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-model-cache\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424543 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-home\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424626 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424558 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-tmp-dir\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.424737 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.424719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-model-cache\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.426634 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.426610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-dshm\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.426836 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.426819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea738c1-cce2-48f2-be41-fd25b306ab54-tls-certs\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.432245 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.432224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wmq\" (UniqueName: \"kubernetes.io/projected/2ea738c1-cce2-48f2-be41-fd25b306ab54-kube-api-access-44wmq\") pod \"auth-test-model-72ce60-kserve-79bd9495-n4wtc\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.480080 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.480045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:06.613557 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:06.613532 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc"] Jul 19 08:34:06.615467 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:34:06.615442 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea738c1_cce2_48f2_be41_fd25b306ab54.slice/crio-9226f16086148e74f21aa93559086402f055e61230b4f763db82253b6c3400d1 WatchSource:0}: Error finding container 9226f16086148e74f21aa93559086402f055e61230b4f763db82253b6c3400d1: Status 404 returned error can't find the container with id 9226f16086148e74f21aa93559086402f055e61230b4f763db82253b6c3400d1 Jul 19 08:34:07.467495 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:07.467457 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" event={"ID":"2ea738c1-cce2-48f2-be41-fd25b306ab54","Type":"ContainerStarted","Data":"580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908"} Jul 19 08:34:07.467931 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:07.467511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" event={"ID":"2ea738c1-cce2-48f2-be41-fd25b306ab54","Type":"ContainerStarted","Data":"9226f16086148e74f21aa93559086402f055e61230b4f763db82253b6c3400d1"} Jul 19 08:34:07.482745 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:07.482694 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" podStartSLOduration=1.482655437 podStartE2EDuration="1.482655437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:07.482069942 +0000 UTC m=+3095.261733256" watchObservedRunningTime="2026-07-19 08:34:07.482655437 +0000 UTC m=+3095.262318758" Jul 19 08:34:10.726773 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:10.726736 2574 scope.go:117] "RemoveContainer" containerID="1e0b2dfe6e62f97bc842f76b18aa3b45e3ae8d62d7184ef428ef08695dd4754c" Jul 19 08:34:11.488538 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:11.488506 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerID="0244eb3693fa66e77da037cfd254e902312d1a1b556388a9dc7641a5f5b30960" exitCode=6 Jul 19 08:34:11.488731 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:11.488579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" event={"ID":"2a4b02bd-e294-47a7-8262-eeb1d906c25d","Type":"ContainerDied","Data":"0244eb3693fa66e77da037cfd254e902312d1a1b556388a9dc7641a5f5b30960"} Jul 19 08:34:11.488731 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:11.488640 2574 scope.go:117] "RemoveContainer" containerID="1e0b2dfe6e62f97bc842f76b18aa3b45e3ae8d62d7184ef428ef08695dd4754c" Jul 19 08:34:11.488999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:11.488968 2574 scope.go:117] "RemoveContainer" containerID="0244eb3693fa66e77da037cfd254e902312d1a1b556388a9dc7641a5f5b30960" Jul 19 08:34:11.489188 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:34:11.489166 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh_odh-ai-gateway-infra(2a4b02bd-e294-47a7-8262-eeb1d906c25d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" Jul 19 08:34:12.513862 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:12.513826 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh"] Jul 19 08:34:12.650197 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:12.650175 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:34:12.788152 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:12.788027 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b02bd-e294-47a7-8262-eeb1d906c25d-maas-api-service-ca\") pod \"2a4b02bd-e294-47a7-8262-eeb1d906c25d\" (UID: \"2a4b02bd-e294-47a7-8262-eeb1d906c25d\") " Jul 19 08:34:12.788406 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:12.788382 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4b02bd-e294-47a7-8262-eeb1d906c25d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a4b02bd-e294-47a7-8262-eeb1d906c25d" (UID: "2a4b02bd-e294-47a7-8262-eeb1d906c25d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:34:12.889450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:12.889417 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b02bd-e294-47a7-8262-eeb1d906c25d-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:34:13.500575 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:13.500542 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" Jul 19 08:34:13.500778 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:13.500577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh" event={"ID":"2a4b02bd-e294-47a7-8262-eeb1d906c25d","Type":"ContainerDied","Data":"549d88ed7275c106d5d721637bd315408e6fc96d58c9024db143eebf6ec3f504"} Jul 19 08:34:13.500778 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:13.500639 2574 scope.go:117] "RemoveContainer" containerID="0244eb3693fa66e77da037cfd254e902312d1a1b556388a9dc7641a5f5b30960" Jul 19 08:34:13.520807 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:13.520779 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh"] Jul 19 08:34:13.532905 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:13.532867 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-f7kjh"] Jul 19 08:34:14.731842 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:14.731805 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" path="/var/lib/kubelet/pods/2a4b02bd-e294-47a7-8262-eeb1d906c25d/volumes" Jul 19 08:34:16.481108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:16.481068 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:16.481514 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:16.481262 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:16.494208 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:16.494183 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:16.527436 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:16.527406 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:34:29.379575 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379541 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng"] Jul 19 08:34:29.380005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379924 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379939 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379958 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379966 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379985 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380005 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.379993 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.380103 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.380116 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.380358 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.380127 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4b02bd-e294-47a7-8262-eeb1d906c25d" containerName="revoke-keys" Jul 19 08:34:29.384912 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.384892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.388396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.388374 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"kube-root-ca.crt\"" Jul 19 08:34:29.388556 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.388427 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"openshift-service-ca.crt\"" Jul 19 08:34:29.388556 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.388455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"default-dockercfg-gdhkn\"" Jul 19 08:34:29.388556 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.388430 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"auth-test-model-190fde-kserve-self-signed-certs\"" Jul 19 08:34:29.391123 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.391101 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng"] Jul 19 08:34:29.441136 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.441102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tls-certs\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.441326 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.441236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tmp-dir\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.441326 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.441289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ms42\" (UniqueName: \"kubernetes.io/projected/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-kube-api-access-9ms42\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.441819 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.441315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-dshm\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.442077 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.442048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-home\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.442161 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.442121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-model-cache\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.543608 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.543567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-model-cache\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.543608 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.543616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tls-certs\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.543883 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.543716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tmp-dir\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.543883 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.543735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ms42\" (UniqueName: \"kubernetes.io/projected/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-kube-api-access-9ms42\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.543883 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.543755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-dshm\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.543883 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.543773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-home\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.544075 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.544022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-model-cache\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.544131 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.544088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-home\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.544131 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.544123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tmp-dir\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.546184 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.546162 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-dshm\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.546342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.546323 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tls-certs\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.552215 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.552189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ms42\" (UniqueName: \"kubernetes.io/projected/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-kube-api-access-9ms42\") pod \"auth-test-model-190fde-kserve-d8c4cf9f5-c54ng\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.695348 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.695256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:29.831214 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:29.831187 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng"] Jul 19 08:34:29.833735 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:34:29.833712 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cbfc819_8cb5_41ec_8c89_8dcbff57b9e8.slice/crio-161d51e72b450e83d8b9a51cae73c661a90dc26ecce2ab5d3806cba78d157ba0 WatchSource:0}: Error finding container 161d51e72b450e83d8b9a51cae73c661a90dc26ecce2ab5d3806cba78d157ba0: Status 404 returned error can't find the container with id 161d51e72b450e83d8b9a51cae73c661a90dc26ecce2ab5d3806cba78d157ba0 Jul 19 08:34:30.582890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:30.582851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" event={"ID":"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8","Type":"ContainerStarted","Data":"35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449"} Jul 19 08:34:30.582890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:30.582898 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" event={"ID":"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8","Type":"ContainerStarted","Data":"161d51e72b450e83d8b9a51cae73c661a90dc26ecce2ab5d3806cba78d157ba0"} Jul 19 08:34:30.600402 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:30.600353 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" podStartSLOduration=1.600335866 podStartE2EDuration="1.600335866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:30.598577666 +0000 UTC m=+3118.378240982" watchObservedRunningTime="2026-07-19 08:34:30.600335866 +0000 UTC m=+3118.379999179" Jul 19 08:34:39.696421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:39.696367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:39.696421 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:39.696437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:39.710295 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:39.710262 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:34:40.647047 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:34:40.647017 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:36:40.655772 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:40.655740 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc"] Jul 19 08:36:41.558413 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:41.558379 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng"] Jul 19 08:36:48.285966 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.285929 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94"] Jul 19 08:36:48.290213 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.290180 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94"] Jul 19 08:36:48.290363 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.290271 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.292592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.292565 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72ce60\"/\"sub-test-model-72ce60-kserve-self-signed-certs\"" Jul 19 08:36:48.384825 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.384785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-tmp-dir\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.385024 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.384834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhfsj\" (UniqueName: \"kubernetes.io/projected/9e860a89-79dc-4962-9750-e7e5edc3f59d-kube-api-access-mhfsj\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.385024 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.384861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-dshm\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.385024 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.384913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-model-cache\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.385024 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.385007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e860a89-79dc-4962-9750-e7e5edc3f59d-tls-certs\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.385198 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.385036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-home\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.485843 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.485803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-model-cache\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486059 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.485896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e860a89-79dc-4962-9750-e7e5edc3f59d-tls-certs\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486059 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.485917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-home\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486059 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.485950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-tmp-dir\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486059 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.485983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhfsj\" (UniqueName: \"kubernetes.io/projected/9e860a89-79dc-4962-9750-e7e5edc3f59d-kube-api-access-mhfsj\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486283 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.486124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-dshm\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486283 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.486272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-model-cache\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486401 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.486309 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-home\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.486461 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.486441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-tmp-dir\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.488388 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.488364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-dshm\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.488552 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.488536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e860a89-79dc-4962-9750-e7e5edc3f59d-tls-certs\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.493425 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.493400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhfsj\" (UniqueName: \"kubernetes.io/projected/9e860a89-79dc-4962-9750-e7e5edc3f59d-kube-api-access-mhfsj\") pod \"sub-test-model-72ce60-kserve-6cf9c45d89-qjd94\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.599783 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.599748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:48.735727 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:48.735704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94"] Jul 19 08:36:48.738016 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:36:48.737989 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e860a89_79dc_4962_9750_e7e5edc3f59d.slice/crio-7edcd90986aa0944f8aec54e5fa3f0e9be1afb6c20697b17e094da62c241e2f2 WatchSource:0}: Error finding container 7edcd90986aa0944f8aec54e5fa3f0e9be1afb6c20697b17e094da62c241e2f2: Status 404 returned error can't find the container with id 7edcd90986aa0944f8aec54e5fa3f0e9be1afb6c20697b17e094da62c241e2f2 Jul 19 08:36:49.248138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:49.248103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" event={"ID":"9e860a89-79dc-4962-9750-e7e5edc3f59d","Type":"ContainerStarted","Data":"5fbe228f806fef0f27feec5dd23325639c4181b024ecc272e1859dd397df829e"} Jul 19 08:36:49.248138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:49.248146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" event={"ID":"9e860a89-79dc-4962-9750-e7e5edc3f59d","Type":"ContainerStarted","Data":"7edcd90986aa0944f8aec54e5fa3f0e9be1afb6c20697b17e094da62c241e2f2"} Jul 19 08:36:49.266749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:49.266667 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" podStartSLOduration=1.26665063 podStartE2EDuration="1.26665063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:36:49.264436304 +0000 UTC m=+3257.044099619" watchObservedRunningTime="2026-07-19 08:36:49.26665063 +0000 UTC m=+3257.046313944" Jul 19 08:36:55.676872 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:55.676833 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" podUID="2ea738c1-cce2-48f2-be41-fd25b306ab54" containerName="main" containerID="cri-o://580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908" gracePeriod=45 Jul 19 08:36:55.929361 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:55.929296 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:36:56.062569 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062537 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-model-cache\") pod \"2ea738c1-cce2-48f2-be41-fd25b306ab54\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " Jul 19 08:36:56.062569 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-tmp-dir\") pod \"2ea738c1-cce2-48f2-be41-fd25b306ab54\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " Jul 19 08:36:56.062841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062606 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-home\") pod \"2ea738c1-cce2-48f2-be41-fd25b306ab54\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " Jul 19 08:36:56.062841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062656 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea738c1-cce2-48f2-be41-fd25b306ab54-tls-certs\") pod \"2ea738c1-cce2-48f2-be41-fd25b306ab54\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " Jul 19 08:36:56.062841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-dshm\") pod \"2ea738c1-cce2-48f2-be41-fd25b306ab54\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " Jul 19 08:36:56.062841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44wmq\" (UniqueName: \"kubernetes.io/projected/2ea738c1-cce2-48f2-be41-fd25b306ab54-kube-api-access-44wmq\") pod \"2ea738c1-cce2-48f2-be41-fd25b306ab54\" (UID: \"2ea738c1-cce2-48f2-be41-fd25b306ab54\") " Jul 19 08:36:56.063067 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062901 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ea738c1-cce2-48f2-be41-fd25b306ab54" (UID: "2ea738c1-cce2-48f2-be41-fd25b306ab54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.063067 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062914 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ea738c1-cce2-48f2-be41-fd25b306ab54" (UID: "2ea738c1-cce2-48f2-be41-fd25b306ab54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.063067 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.062925 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-home" (OuterVolumeSpecName: "home") pod "2ea738c1-cce2-48f2-be41-fd25b306ab54" (UID: "2ea738c1-cce2-48f2-be41-fd25b306ab54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.063194 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.063111 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.063194 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.063123 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.063194 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.063138 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.065095 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.065073 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-dshm" (OuterVolumeSpecName: "dshm") pod "2ea738c1-cce2-48f2-be41-fd25b306ab54" (UID: "2ea738c1-cce2-48f2-be41-fd25b306ab54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.065190 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.065170 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea738c1-cce2-48f2-be41-fd25b306ab54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ea738c1-cce2-48f2-be41-fd25b306ab54" (UID: "2ea738c1-cce2-48f2-be41-fd25b306ab54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:36:56.065257 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.065241 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea738c1-cce2-48f2-be41-fd25b306ab54-kube-api-access-44wmq" (OuterVolumeSpecName: "kube-api-access-44wmq") pod "2ea738c1-cce2-48f2-be41-fd25b306ab54" (UID: "2ea738c1-cce2-48f2-be41-fd25b306ab54"). InnerVolumeSpecName "kube-api-access-44wmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:36:56.163869 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.163834 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea738c1-cce2-48f2-be41-fd25b306ab54-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.163869 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.163862 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea738c1-cce2-48f2-be41-fd25b306ab54-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.163869 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.163874 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44wmq\" (UniqueName: \"kubernetes.io/projected/2ea738c1-cce2-48f2-be41-fd25b306ab54-kube-api-access-44wmq\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.285138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.285044 2574 generic.go:356] "Generic (PLEG): container finished" podID="2ea738c1-cce2-48f2-be41-fd25b306ab54" containerID="580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908" exitCode=0 Jul 19 08:36:56.285138 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.285093 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" Jul 19 08:36:56.285363 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.285134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" event={"ID":"2ea738c1-cce2-48f2-be41-fd25b306ab54","Type":"ContainerDied","Data":"580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908"} Jul 19 08:36:56.285363 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.285177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc" event={"ID":"2ea738c1-cce2-48f2-be41-fd25b306ab54","Type":"ContainerDied","Data":"9226f16086148e74f21aa93559086402f055e61230b4f763db82253b6c3400d1"} Jul 19 08:36:56.285363 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.285194 2574 scope.go:117] "RemoveContainer" containerID="580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908" Jul 19 08:36:56.294183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.294164 2574 scope.go:117] "RemoveContainer" containerID="580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908" Jul 19 08:36:56.294429 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:36:56.294414 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908\": container with ID starting with 580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908 not found: ID does not exist" containerID="580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908" Jul 19 08:36:56.294468 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.294436 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908"} err="failed to get container status \"580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908\": rpc error: code = NotFound desc = could not find container \"580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908\": container with ID starting with 580297d708a3ad7b619197ada010befb30fc9ac39d72a88fd026b750d48c9908 not found: ID does not exist" Jul 19 08:36:56.305827 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.305801 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc"] Jul 19 08:36:56.309022 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.309001 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/auth-test-model-72ce60-kserve-79bd9495-n4wtc"] Jul 19 08:36:56.579291 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.579194 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" podUID="8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" containerName="main" containerID="cri-o://35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449" gracePeriod=45 Jul 19 08:36:56.732739 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.732699 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea738c1-cce2-48f2-be41-fd25b306ab54" path="/var/lib/kubelet/pods/2ea738c1-cce2-48f2-be41-fd25b306ab54/volumes" Jul 19 08:36:56.838250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.838228 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:36:56.971970 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.971921 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-dshm\") pod \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " Jul 19 08:36:56.971970 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.971973 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-home\") pod \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " Jul 19 08:36:56.972250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972002 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ms42\" (UniqueName: \"kubernetes.io/projected/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-kube-api-access-9ms42\") pod \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " Jul 19 08:36:56.972250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972034 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-model-cache\") pod \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " Jul 19 08:36:56.972250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972133 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tmp-dir\") pod \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " Jul 19 08:36:56.972250 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972162 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tls-certs\") pod \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\" (UID: \"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8\") " Jul 19 08:36:56.972472 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972256 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-home" (OuterVolumeSpecName: "home") pod "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" (UID: "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.972472 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972391 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" (UID: "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.972472 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972434 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" (UID: "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.972625 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972496 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.972625 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.972515 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:56.974474 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.974449 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-dshm" (OuterVolumeSpecName: "dshm") pod "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" (UID: "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:36:56.974587 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.974473 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" (UID: "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:36:56.974587 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:56.974472 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-kube-api-access-9ms42" (OuterVolumeSpecName: "kube-api-access-9ms42") pod "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" (UID: "8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8"). InnerVolumeSpecName "kube-api-access-9ms42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:36:57.073457 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.073405 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:57.073457 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.073449 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:57.073457 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.073461 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ms42\" (UniqueName: \"kubernetes.io/projected/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-kube-api-access-9ms42\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:57.073457 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.073474 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:36:57.292518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.292430 2574 generic.go:356] "Generic (PLEG): container finished" podID="8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" containerID="35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449" exitCode=0 Jul 19 08:36:57.292518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.292474 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" Jul 19 08:36:57.292518 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.292503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" event={"ID":"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8","Type":"ContainerDied","Data":"35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449"} Jul 19 08:36:57.292818 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.292537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng" event={"ID":"8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8","Type":"ContainerDied","Data":"161d51e72b450e83d8b9a51cae73c661a90dc26ecce2ab5d3806cba78d157ba0"} Jul 19 08:36:57.292818 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.292559 2574 scope.go:117] "RemoveContainer" containerID="35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449" Jul 19 08:36:57.301979 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.301960 2574 scope.go:117] "RemoveContainer" containerID="35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449" Jul 19 08:36:57.302262 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:36:57.302241 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449\": container with ID starting with 35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449 not found: ID does not exist" containerID="35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449" Jul 19 08:36:57.302315 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.302272 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449"} err="failed to get container status \"35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449\": rpc error: code = NotFound desc = could not find container \"35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449\": container with ID starting with 35b97d8d05d4c6ee505783bca8358934bb4d086b5ae6350789c256e256240449 not found: ID does not exist" Jul 19 08:36:57.312859 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.312824 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng"] Jul 19 08:36:57.317154 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:57.317128 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/auth-test-model-190fde-kserve-d8c4cf9f5-c54ng"] Jul 19 08:36:58.600082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:58.600045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:58.600492 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:58.600097 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:58.612523 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:58.612492 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:36:58.731799 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:58.731764 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" path="/var/lib/kubelet/pods/8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8/volumes" Jul 19 08:36:59.314800 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:36:59.314771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:37:18.278381 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278337 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5"] Jul 19 08:37:18.278915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278743 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" containerName="main" Jul 19 08:37:18.278915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278757 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" containerName="main" Jul 19 08:37:18.278915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278776 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea738c1-cce2-48f2-be41-fd25b306ab54" containerName="main" Jul 19 08:37:18.278915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278782 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea738c1-cce2-48f2-be41-fd25b306ab54" containerName="main" Jul 19 08:37:18.278915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278845 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbfc819-8cb5-41ec-8c89-8dcbff57b9e8" containerName="main" Jul 19 08:37:18.278915 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.278851 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea738c1-cce2-48f2-be41-fd25b306ab54" containerName="main" Jul 19 08:37:18.281749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.281730 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.290544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.290304 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"openshift-service-ca.crt\"" Jul 19 08:37:18.290544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.290370 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"default-dockercfg-gdhkn\"" Jul 19 08:37:18.290544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.290379 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"sub-test-model-190fde-kserve-self-signed-certs\"" Jul 19 08:37:18.290837 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.290602 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"kube-root-ca.crt\"" Jul 19 08:37:18.291209 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.291184 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5"] Jul 19 08:37:18.382840 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.382800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b452e033-b300-4114-874e-8051e8f8b2f3-tls-certs\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.383065 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.382931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-model-cache\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.383065 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.382977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-home\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.383065 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.383030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-tmp-dir\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.383185 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.383088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-dshm\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.383185 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.383131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmwq\" (UniqueName: \"kubernetes.io/projected/b452e033-b300-4114-874e-8051e8f8b2f3-kube-api-access-sdmwq\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.483928 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.483890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-model-cache\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.483928 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.483929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-home\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.483958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-tmp-dir\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.483976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-dshm\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.484008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmwq\" (UniqueName: \"kubernetes.io/projected/b452e033-b300-4114-874e-8051e8f8b2f3-kube-api-access-sdmwq\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484173 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.484039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b452e033-b300-4114-874e-8051e8f8b2f3-tls-certs\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.484323 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-model-cache\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484396 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.484373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-home\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.484493 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.484411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-tmp-dir\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.486557 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.486527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-dshm\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.486667 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.486635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b452e033-b300-4114-874e-8051e8f8b2f3-tls-certs\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.491896 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.491878 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmwq\" (UniqueName: \"kubernetes.io/projected/b452e033-b300-4114-874e-8051e8f8b2f3-kube-api-access-sdmwq\") pod \"sub-test-model-190fde-kserve-cbd8d86cc-hr6t5\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.593858 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.593830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:18.733620 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:18.733592 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5"] Jul 19 08:37:18.735640 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:37:18.735618 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb452e033_b300_4114_874e_8051e8f8b2f3.slice/crio-7c17ef9c05d649bf2119cb66e4a05c1a27c5ea8f726d0585e4e1e02b972a21d9 WatchSource:0}: Error finding container 7c17ef9c05d649bf2119cb66e4a05c1a27c5ea8f726d0585e4e1e02b972a21d9: Status 404 returned error can't find the container with id 7c17ef9c05d649bf2119cb66e4a05c1a27c5ea8f726d0585e4e1e02b972a21d9 Jul 19 08:37:19.400971 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:19.400933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" event={"ID":"b452e033-b300-4114-874e-8051e8f8b2f3","Type":"ContainerStarted","Data":"90719b6f5bf3b62688e775fd351f7b6c79e7222dd339a8ea8fd3634fb32a5056"} Jul 19 08:37:19.401387 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:19.400979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" event={"ID":"b452e033-b300-4114-874e-8051e8f8b2f3","Type":"ContainerStarted","Data":"7c17ef9c05d649bf2119cb66e4a05c1a27c5ea8f726d0585e4e1e02b972a21d9"} Jul 19 08:37:19.417280 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:19.417220 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" podStartSLOduration=1.417203186 podStartE2EDuration="1.417203186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:37:19.4153113 +0000 UTC m=+3287.194974616" watchObservedRunningTime="2026-07-19 08:37:19.417203186 +0000 UTC m=+3287.196866500" Jul 19 08:37:28.594616 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:28.594568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:28.595170 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:28.594632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:28.607730 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:28.607702 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:37:29.461540 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:37:29.461509 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:38:09.509873 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:09.509777 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94"] Jul 19 08:38:10.486452 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:10.486417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5"] Jul 19 08:38:17.076538 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.076501 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl"] Jul 19 08:38:17.080040 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.080024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.082253 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.082232 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72ce60\"/\"rate-test-model-72ce60-kserve-self-signed-certs\"" Jul 19 08:38:17.089826 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.089803 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl"] Jul 19 08:38:17.259841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.259804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-home\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.259841 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.259847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-dshm\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.260088 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.259917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-tmp-dir\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.260088 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.259946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzd7\" (UniqueName: \"kubernetes.io/projected/f0eb9302-643f-4535-999e-258fca42136d-kube-api-access-7qzd7\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.260088 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.259972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb9302-643f-4535-999e-258fca42136d-tls-certs\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.260088 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.260004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-model-cache\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.360810 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.360774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-home\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.360999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.360824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-dshm\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.360999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.360864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-tmp-dir\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.360999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.360900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzd7\" (UniqueName: \"kubernetes.io/projected/f0eb9302-643f-4535-999e-258fca42136d-kube-api-access-7qzd7\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.360999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.360936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb9302-643f-4535-999e-258fca42136d-tls-certs\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.360999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.360988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-model-cache\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.361277 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.361224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-home\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.361341 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.361293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-tmp-dir\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.361412 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.361394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-model-cache\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.363318 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.363297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-dshm\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.363746 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.363725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb9302-643f-4535-999e-258fca42136d-tls-certs\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.368843 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.368815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzd7\" (UniqueName: \"kubernetes.io/projected/f0eb9302-643f-4535-999e-258fca42136d-kube-api-access-7qzd7\") pod \"rate-test-model-72ce60-kserve-6f7df8c579-p8vgl\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.390274 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.390234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:17.533072 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.533046 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl"] Jul 19 08:38:17.535656 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:38:17.535618 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0eb9302_643f_4535_999e_258fca42136d.slice/crio-8b89426ebc319de5c40ea5874301c798397b190d78b6c2707958ef22de539887 WatchSource:0}: Error finding container 8b89426ebc319de5c40ea5874301c798397b190d78b6c2707958ef22de539887: Status 404 returned error can't find the container with id 8b89426ebc319de5c40ea5874301c798397b190d78b6c2707958ef22de539887 Jul 19 08:38:17.688876 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.688784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" event={"ID":"f0eb9302-643f-4535-999e-258fca42136d","Type":"ContainerStarted","Data":"2be45ce77255559111d7c394653c5d9f54f15c00ffaf60fd000ffb7e4a89a8ec"} Jul 19 08:38:17.688876 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.688828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" event={"ID":"f0eb9302-643f-4535-999e-258fca42136d","Type":"ContainerStarted","Data":"8b89426ebc319de5c40ea5874301c798397b190d78b6c2707958ef22de539887"} Jul 19 08:38:17.710955 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:17.710895 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" podStartSLOduration=0.710878628 podStartE2EDuration="710.878628ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:38:17.708786977 +0000 UTC m=+3345.488450299" watchObservedRunningTime="2026-07-19 08:38:17.710878628 +0000 UTC m=+3345.490541941" Jul 19 08:38:24.530831 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.530790 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" podUID="9e860a89-79dc-4962-9750-e7e5edc3f59d" containerName="main" containerID="cri-o://5fbe228f806fef0f27feec5dd23325639c4181b024ecc272e1859dd397df829e" gracePeriod=45 Jul 19 08:38:24.723614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.723578 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e860a89-79dc-4962-9750-e7e5edc3f59d" containerID="5fbe228f806fef0f27feec5dd23325639c4181b024ecc272e1859dd397df829e" exitCode=0 Jul 19 08:38:24.723614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.723628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" event={"ID":"9e860a89-79dc-4962-9750-e7e5edc3f59d","Type":"ContainerDied","Data":"5fbe228f806fef0f27feec5dd23325639c4181b024ecc272e1859dd397df829e"} Jul 19 08:38:24.781827 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.781762 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:38:24.831885 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.831854 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-dshm\") pod \"9e860a89-79dc-4962-9750-e7e5edc3f59d\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " Jul 19 08:38:24.832053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.831947 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e860a89-79dc-4962-9750-e7e5edc3f59d-tls-certs\") pod \"9e860a89-79dc-4962-9750-e7e5edc3f59d\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " Jul 19 08:38:24.832053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.831974 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-home\") pod \"9e860a89-79dc-4962-9750-e7e5edc3f59d\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " Jul 19 08:38:24.832053 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832020 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhfsj\" (UniqueName: \"kubernetes.io/projected/9e860a89-79dc-4962-9750-e7e5edc3f59d-kube-api-access-mhfsj\") pod \"9e860a89-79dc-4962-9750-e7e5edc3f59d\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " Jul 19 08:38:24.832245 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-tmp-dir\") pod \"9e860a89-79dc-4962-9750-e7e5edc3f59d\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " Jul 19 08:38:24.832245 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832125 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-model-cache\") pod \"9e860a89-79dc-4962-9750-e7e5edc3f59d\" (UID: \"9e860a89-79dc-4962-9750-e7e5edc3f59d\") " Jul 19 08:38:24.832352 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832256 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-home" (OuterVolumeSpecName: "home") pod "9e860a89-79dc-4962-9750-e7e5edc3f59d" (UID: "9e860a89-79dc-4962-9750-e7e5edc3f59d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:24.832419 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832349 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e860a89-79dc-4962-9750-e7e5edc3f59d" (UID: "9e860a89-79dc-4962-9750-e7e5edc3f59d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:24.832540 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832520 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e860a89-79dc-4962-9750-e7e5edc3f59d" (UID: "9e860a89-79dc-4962-9750-e7e5edc3f59d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:24.832743 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832643 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:24.832743 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832695 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:24.832743 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.832710 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:24.834300 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.834265 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-dshm" (OuterVolumeSpecName: "dshm") pod "9e860a89-79dc-4962-9750-e7e5edc3f59d" (UID: "9e860a89-79dc-4962-9750-e7e5edc3f59d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:24.834300 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.834284 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e860a89-79dc-4962-9750-e7e5edc3f59d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e860a89-79dc-4962-9750-e7e5edc3f59d" (UID: "9e860a89-79dc-4962-9750-e7e5edc3f59d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:38:24.834653 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.834631 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e860a89-79dc-4962-9750-e7e5edc3f59d-kube-api-access-mhfsj" (OuterVolumeSpecName: "kube-api-access-mhfsj") pod "9e860a89-79dc-4962-9750-e7e5edc3f59d" (UID: "9e860a89-79dc-4962-9750-e7e5edc3f59d"). InnerVolumeSpecName "kube-api-access-mhfsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:38:24.934082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.934045 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e860a89-79dc-4962-9750-e7e5edc3f59d-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:24.934082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.934078 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhfsj\" (UniqueName: \"kubernetes.io/projected/9e860a89-79dc-4962-9750-e7e5edc3f59d-kube-api-access-mhfsj\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:24.934082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:24.934088 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e860a89-79dc-4962-9750-e7e5edc3f59d-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:25.508240 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.508178 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" podUID="b452e033-b300-4114-874e-8051e8f8b2f3" containerName="main" containerID="cri-o://90719b6f5bf3b62688e775fd351f7b6c79e7222dd339a8ea8fd3634fb32a5056" gracePeriod=45 Jul 19 08:38:25.729930 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.729897 2574 generic.go:356] "Generic (PLEG): container finished" podID="b452e033-b300-4114-874e-8051e8f8b2f3" containerID="90719b6f5bf3b62688e775fd351f7b6c79e7222dd339a8ea8fd3634fb32a5056" exitCode=0 Jul 19 08:38:25.730336 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.729976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" event={"ID":"b452e033-b300-4114-874e-8051e8f8b2f3","Type":"ContainerDied","Data":"90719b6f5bf3b62688e775fd351f7b6c79e7222dd339a8ea8fd3634fb32a5056"} Jul 19 08:38:25.731311 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.731285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" event={"ID":"9e860a89-79dc-4962-9750-e7e5edc3f59d","Type":"ContainerDied","Data":"7edcd90986aa0944f8aec54e5fa3f0e9be1afb6c20697b17e094da62c241e2f2"} Jul 19 08:38:25.731457 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.731322 2574 scope.go:117] "RemoveContainer" containerID="5fbe228f806fef0f27feec5dd23325639c4181b024ecc272e1859dd397df829e" Jul 19 08:38:25.731457 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.731293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94" Jul 19 08:38:25.753167 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.753140 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94"] Jul 19 08:38:25.754911 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.754886 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/sub-test-model-72ce60-kserve-6cf9c45d89-qjd94"] Jul 19 08:38:25.778404 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.778381 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:38:25.843892 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.843858 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b452e033-b300-4114-874e-8051e8f8b2f3-tls-certs\") pod \"b452e033-b300-4114-874e-8051e8f8b2f3\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " Jul 19 08:38:25.844089 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.843913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdmwq\" (UniqueName: \"kubernetes.io/projected/b452e033-b300-4114-874e-8051e8f8b2f3-kube-api-access-sdmwq\") pod \"b452e033-b300-4114-874e-8051e8f8b2f3\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " Jul 19 08:38:25.844089 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.843962 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-model-cache\") pod \"b452e033-b300-4114-874e-8051e8f8b2f3\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " Jul 19 08:38:25.844224 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844129 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-tmp-dir\") pod \"b452e033-b300-4114-874e-8051e8f8b2f3\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " Jul 19 08:38:25.844283 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844226 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-home\") pod \"b452e033-b300-4114-874e-8051e8f8b2f3\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " Jul 19 08:38:25.844283 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844266 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-dshm\") pod \"b452e033-b300-4114-874e-8051e8f8b2f3\" (UID: \"b452e033-b300-4114-874e-8051e8f8b2f3\") " Jul 19 08:38:25.844389 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844222 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b452e033-b300-4114-874e-8051e8f8b2f3" (UID: "b452e033-b300-4114-874e-8051e8f8b2f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:25.844448 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844392 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b452e033-b300-4114-874e-8051e8f8b2f3" (UID: "b452e033-b300-4114-874e-8051e8f8b2f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:25.844448 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844404 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-home" (OuterVolumeSpecName: "home") pod "b452e033-b300-4114-874e-8051e8f8b2f3" (UID: "b452e033-b300-4114-874e-8051e8f8b2f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:25.844656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844605 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:25.844656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844626 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:25.844656 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.844637 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:25.846228 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.846203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b452e033-b300-4114-874e-8051e8f8b2f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b452e033-b300-4114-874e-8051e8f8b2f3" (UID: "b452e033-b300-4114-874e-8051e8f8b2f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:38:25.846333 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.846238 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b452e033-b300-4114-874e-8051e8f8b2f3-kube-api-access-sdmwq" (OuterVolumeSpecName: "kube-api-access-sdmwq") pod "b452e033-b300-4114-874e-8051e8f8b2f3" (UID: "b452e033-b300-4114-874e-8051e8f8b2f3"). InnerVolumeSpecName "kube-api-access-sdmwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:38:25.846333 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.846264 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-dshm" (OuterVolumeSpecName: "dshm") pod "b452e033-b300-4114-874e-8051e8f8b2f3" (UID: "b452e033-b300-4114-874e-8051e8f8b2f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:38:25.945699 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.945632 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b452e033-b300-4114-874e-8051e8f8b2f3-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:25.945699 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.945664 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmwq\" (UniqueName: \"kubernetes.io/projected/b452e033-b300-4114-874e-8051e8f8b2f3-kube-api-access-sdmwq\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:25.945917 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:25.945715 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b452e033-b300-4114-874e-8051e8f8b2f3-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:38:26.733579 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:26.733544 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e860a89-79dc-4962-9750-e7e5edc3f59d" path="/var/lib/kubelet/pods/9e860a89-79dc-4962-9750-e7e5edc3f59d/volumes" Jul 19 08:38:26.737971 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:26.737949 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" Jul 19 08:38:26.738108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:26.738001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5" event={"ID":"b452e033-b300-4114-874e-8051e8f8b2f3","Type":"ContainerDied","Data":"7c17ef9c05d649bf2119cb66e4a05c1a27c5ea8f726d0585e4e1e02b972a21d9"} Jul 19 08:38:26.738108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:26.738062 2574 scope.go:117] "RemoveContainer" containerID="90719b6f5bf3b62688e775fd351f7b6c79e7222dd339a8ea8fd3634fb32a5056" Jul 19 08:38:26.758146 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:26.758117 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5"] Jul 19 08:38:26.764215 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:26.764192 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/sub-test-model-190fde-kserve-cbd8d86cc-hr6t5"] Jul 19 08:38:27.390822 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:27.390767 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:27.390998 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:27.390842 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:27.403428 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:27.403398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:27.756262 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:27.756177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:38:28.733384 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:38:28.733345 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b452e033-b300-4114-874e-8051e8f8b2f3" path="/var/lib/kubelet/pods/b452e033-b300-4114-874e-8051e8f8b2f3/volumes" Jul 19 08:39:03.378731 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.378668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn"] Jul 19 08:39:03.379299 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.379278 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b452e033-b300-4114-874e-8051e8f8b2f3" containerName="main" Jul 19 08:39:03.379369 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.379302 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b452e033-b300-4114-874e-8051e8f8b2f3" containerName="main" Jul 19 08:39:03.379369 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.379315 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e860a89-79dc-4962-9750-e7e5edc3f59d" containerName="main" Jul 19 08:39:03.379369 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.379324 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e860a89-79dc-4962-9750-e7e5edc3f59d" containerName="main" Jul 19 08:39:03.379467 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.379416 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e860a89-79dc-4962-9750-e7e5edc3f59d" containerName="main" Jul 19 08:39:03.379467 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.379428 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b452e033-b300-4114-874e-8051e8f8b2f3" containerName="main" Jul 19 08:39:03.382700 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.382661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.386320 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.386298 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"rate-test-model-190fde-kserve-self-signed-certs\"" Jul 19 08:39:03.386438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.386300 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"default-dockercfg-gdhkn\"" Jul 19 08:39:03.386438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.386300 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"openshift-service-ca.crt\"" Jul 19 08:39:03.386438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.386359 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-190fde\"/\"kube-root-ca.crt\"" Jul 19 08:39:03.391401 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.391377 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn"] Jul 19 08:39:03.404769 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.404744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-dshm\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.404916 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.404808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-model-cache\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.404916 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.404827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488de101-d58f-485e-8a97-8dd1e66b1cdd-tls-certs\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.404916 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.404851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggnl\" (UniqueName: \"kubernetes.io/projected/488de101-d58f-485e-8a97-8dd1e66b1cdd-kube-api-access-4ggnl\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.405106 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.404918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-tmp-dir\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.405106 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.404991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-home\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-model-cache\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506203 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488de101-d58f-485e-8a97-8dd1e66b1cdd-tls-certs\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506471 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggnl\" (UniqueName: \"kubernetes.io/projected/488de101-d58f-485e-8a97-8dd1e66b1cdd-kube-api-access-4ggnl\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506471 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-tmp-dir\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506471 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-home\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506471 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-dshm\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506742 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-model-cache\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506742 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-tmp-dir\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.506853 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.506775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-home\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.508758 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.508738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-dshm\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.508905 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.508888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488de101-d58f-485e-8a97-8dd1e66b1cdd-tls-certs\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.513903 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.513887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggnl\" (UniqueName: \"kubernetes.io/projected/488de101-d58f-485e-8a97-8dd1e66b1cdd-kube-api-access-4ggnl\") pod \"rate-test-model-190fde-kserve-cf9558bff-qt4kn\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.692919 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.692829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:03.828108 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.828080 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn"] Jul 19 08:39:03.830616 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:39:03.830585 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod488de101_d58f_485e_8a97_8dd1e66b1cdd.slice/crio-b9cc2a6c7b250342f8d93ba15a36fe945190238505ebc729dfd2511142570795 WatchSource:0}: Error finding container b9cc2a6c7b250342f8d93ba15a36fe945190238505ebc729dfd2511142570795: Status 404 returned error can't find the container with id b9cc2a6c7b250342f8d93ba15a36fe945190238505ebc729dfd2511142570795 Jul 19 08:39:03.910531 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.910493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" event={"ID":"488de101-d58f-485e-8a97-8dd1e66b1cdd","Type":"ContainerStarted","Data":"da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6"} Jul 19 08:39:03.910956 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.910545 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" event={"ID":"488de101-d58f-485e-8a97-8dd1e66b1cdd","Type":"ContainerStarted","Data":"b9cc2a6c7b250342f8d93ba15a36fe945190238505ebc729dfd2511142570795"} Jul 19 08:39:03.926415 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:03.926347 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" podStartSLOduration=0.92632914 podStartE2EDuration="926.32914ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:39:03.924821212 +0000 UTC m=+3391.704484528" watchObservedRunningTime="2026-07-19 08:39:03.92632914 +0000 UTC m=+3391.705992454" Jul 19 08:39:13.694055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:13.694006 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:13.694458 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:13.694070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:13.707077 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:13.707051 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:13.969693 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:13.969577 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:39:52.053098 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:52.053064 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl"] Jul 19 08:39:52.901890 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:39:52.901856 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn"] Jul 19 08:40:07.073600 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.073527 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" podUID="f0eb9302-643f-4535-999e-258fca42136d" containerName="main" containerID="cri-o://2be45ce77255559111d7c394653c5d9f54f15c00ffaf60fd000ffb7e4a89a8ec" gracePeriod=45 Jul 19 08:40:07.213443 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.213413 2574 generic.go:356] "Generic (PLEG): container finished" podID="f0eb9302-643f-4535-999e-258fca42136d" containerID="2be45ce77255559111d7c394653c5d9f54f15c00ffaf60fd000ffb7e4a89a8ec" exitCode=0 Jul 19 08:40:07.213613 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.213492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" event={"ID":"f0eb9302-643f-4535-999e-258fca42136d","Type":"ContainerDied","Data":"2be45ce77255559111d7c394653c5d9f54f15c00ffaf60fd000ffb7e4a89a8ec"} Jul 19 08:40:07.337986 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.337964 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:40:07.430739 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.430672 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-home\") pod \"f0eb9302-643f-4535-999e-258fca42136d\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " Jul 19 08:40:07.430929 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.430752 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-tmp-dir\") pod \"f0eb9302-643f-4535-999e-258fca42136d\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " Jul 19 08:40:07.430929 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.430824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-dshm\") pod \"f0eb9302-643f-4535-999e-258fca42136d\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " Jul 19 08:40:07.430929 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.430856 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-model-cache\") pod \"f0eb9302-643f-4535-999e-258fca42136d\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " Jul 19 08:40:07.430929 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.430895 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb9302-643f-4535-999e-258fca42136d-tls-certs\") pod \"f0eb9302-643f-4535-999e-258fca42136d\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " Jul 19 08:40:07.430929 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.430927 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qzd7\" (UniqueName: \"kubernetes.io/projected/f0eb9302-643f-4535-999e-258fca42136d-kube-api-access-7qzd7\") pod \"f0eb9302-643f-4535-999e-258fca42136d\" (UID: \"f0eb9302-643f-4535-999e-258fca42136d\") " Jul 19 08:40:07.431202 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.431024 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-home" (OuterVolumeSpecName: "home") pod "f0eb9302-643f-4535-999e-258fca42136d" (UID: "f0eb9302-643f-4535-999e-258fca42136d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:07.431202 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.431031 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0eb9302-643f-4535-999e-258fca42136d" (UID: "f0eb9302-643f-4535-999e-258fca42136d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:07.431202 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.431175 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0eb9302-643f-4535-999e-258fca42136d" (UID: "f0eb9302-643f-4535-999e-258fca42136d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:07.431338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.431251 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:07.431338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.431281 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:07.433208 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.433174 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-dshm" (OuterVolumeSpecName: "dshm") pod "f0eb9302-643f-4535-999e-258fca42136d" (UID: "f0eb9302-643f-4535-999e-258fca42136d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:07.433208 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.433192 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0eb9302-643f-4535-999e-258fca42136d-kube-api-access-7qzd7" (OuterVolumeSpecName: "kube-api-access-7qzd7") pod "f0eb9302-643f-4535-999e-258fca42136d" (UID: "f0eb9302-643f-4535-999e-258fca42136d"). InnerVolumeSpecName "kube-api-access-7qzd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:07.433350 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.433209 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0eb9302-643f-4535-999e-258fca42136d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0eb9302-643f-4535-999e-258fca42136d" (UID: "f0eb9302-643f-4535-999e-258fca42136d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:40:07.531914 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.531875 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzd7\" (UniqueName: \"kubernetes.io/projected/f0eb9302-643f-4535-999e-258fca42136d-kube-api-access-7qzd7\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:07.531914 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.531905 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:07.531914 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.531915 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0eb9302-643f-4535-999e-258fca42136d-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:07.531914 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.531924 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0eb9302-643f-4535-999e-258fca42136d-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:07.922504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:07.922467 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" podUID="488de101-d58f-485e-8a97-8dd1e66b1cdd" containerName="main" containerID="cri-o://da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6" gracePeriod=45 Jul 19 08:40:08.176562 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.176497 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:40:08.219799 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.219760 2574 generic.go:356] "Generic (PLEG): container finished" podID="488de101-d58f-485e-8a97-8dd1e66b1cdd" containerID="da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6" exitCode=0 Jul 19 08:40:08.219799 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.219806 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" Jul 19 08:40:08.220063 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.219841 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" event={"ID":"488de101-d58f-485e-8a97-8dd1e66b1cdd","Type":"ContainerDied","Data":"da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6"} Jul 19 08:40:08.220063 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.219879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn" event={"ID":"488de101-d58f-485e-8a97-8dd1e66b1cdd","Type":"ContainerDied","Data":"b9cc2a6c7b250342f8d93ba15a36fe945190238505ebc729dfd2511142570795"} Jul 19 08:40:08.220063 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.219899 2574 scope.go:117] "RemoveContainer" containerID="da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6" Jul 19 08:40:08.221249 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.221230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" event={"ID":"f0eb9302-643f-4535-999e-258fca42136d","Type":"ContainerDied","Data":"8b89426ebc319de5c40ea5874301c798397b190d78b6c2707958ef22de539887"} Jul 19 08:40:08.221249 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.221242 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl" Jul 19 08:40:08.229016 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.228906 2574 scope.go:117] "RemoveContainer" containerID="da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6" Jul 19 08:40:08.229294 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:40:08.229253 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6\": container with ID starting with da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6 not found: ID does not exist" containerID="da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6" Jul 19 08:40:08.229413 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.229293 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6"} err="failed to get container status \"da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6\": rpc error: code = NotFound desc = could not find container \"da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6\": container with ID starting with da2c7cb293e9b4075c7df68f9580b5033c09c92d19c606dc4ec62e4be83abcc6 not found: ID does not exist" Jul 19 08:40:08.229413 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.229311 2574 scope.go:117] "RemoveContainer" containerID="2be45ce77255559111d7c394653c5d9f54f15c00ffaf60fd000ffb7e4a89a8ec" Jul 19 08:40:08.242006 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.241982 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl"] Jul 19 08:40:08.246034 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.246011 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-72ce60/rate-test-model-72ce60-kserve-6f7df8c579-p8vgl"] Jul 19 08:40:08.340094 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-home\") pod \"488de101-d58f-485e-8a97-8dd1e66b1cdd\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " Jul 19 08:40:08.340094 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340098 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-model-cache\") pod \"488de101-d58f-485e-8a97-8dd1e66b1cdd\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " Jul 19 08:40:08.340345 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340131 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488de101-d58f-485e-8a97-8dd1e66b1cdd-tls-certs\") pod \"488de101-d58f-485e-8a97-8dd1e66b1cdd\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " Jul 19 08:40:08.340345 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340173 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ggnl\" (UniqueName: \"kubernetes.io/projected/488de101-d58f-485e-8a97-8dd1e66b1cdd-kube-api-access-4ggnl\") pod \"488de101-d58f-485e-8a97-8dd1e66b1cdd\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " Jul 19 08:40:08.340345 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340241 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-tmp-dir\") pod \"488de101-d58f-485e-8a97-8dd1e66b1cdd\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " Jul 19 08:40:08.340345 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-dshm\") pod \"488de101-d58f-485e-8a97-8dd1e66b1cdd\" (UID: \"488de101-d58f-485e-8a97-8dd1e66b1cdd\") " Jul 19 08:40:08.340561 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340342 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-home" (OuterVolumeSpecName: "home") pod "488de101-d58f-485e-8a97-8dd1e66b1cdd" (UID: "488de101-d58f-485e-8a97-8dd1e66b1cdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:08.340561 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340435 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "488de101-d58f-485e-8a97-8dd1e66b1cdd" (UID: "488de101-d58f-485e-8a97-8dd1e66b1cdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:08.340561 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340492 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "488de101-d58f-485e-8a97-8dd1e66b1cdd" (UID: "488de101-d58f-485e-8a97-8dd1e66b1cdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:08.340783 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340756 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-tmp-dir\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:08.340783 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340783 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-home\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:08.340953 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.340797 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-model-cache\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:08.342484 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.342458 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-dshm" (OuterVolumeSpecName: "dshm") pod "488de101-d58f-485e-8a97-8dd1e66b1cdd" (UID: "488de101-d58f-485e-8a97-8dd1e66b1cdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 08:40:08.342575 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.342491 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488de101-d58f-485e-8a97-8dd1e66b1cdd-kube-api-access-4ggnl" (OuterVolumeSpecName: "kube-api-access-4ggnl") pod "488de101-d58f-485e-8a97-8dd1e66b1cdd" (UID: "488de101-d58f-485e-8a97-8dd1e66b1cdd"). InnerVolumeSpecName "kube-api-access-4ggnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:08.342575 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.342510 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488de101-d58f-485e-8a97-8dd1e66b1cdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "488de101-d58f-485e-8a97-8dd1e66b1cdd" (UID: "488de101-d58f-485e-8a97-8dd1e66b1cdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:40:08.441913 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.441825 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488de101-d58f-485e-8a97-8dd1e66b1cdd-dshm\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:08.441913 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.441853 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488de101-d58f-485e-8a97-8dd1e66b1cdd-tls-certs\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:08.441913 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.441864 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggnl\" (UniqueName: \"kubernetes.io/projected/488de101-d58f-485e-8a97-8dd1e66b1cdd-kube-api-access-4ggnl\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:08.541010 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.540970 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn"] Jul 19 08:40:08.544303 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.544276 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-190fde/rate-test-model-190fde-kserve-cf9558bff-qt4kn"] Jul 19 08:40:08.732079 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.731994 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488de101-d58f-485e-8a97-8dd1e66b1cdd" path="/var/lib/kubelet/pods/488de101-d58f-485e-8a97-8dd1e66b1cdd/volumes" Jul 19 08:40:08.732406 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:08.732392 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0eb9302-643f-4535-999e-258fca42136d" path="/var/lib/kubelet/pods/f0eb9302-643f-4535-999e-258fca42136d/volumes" Jul 19 08:40:25.683190 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683154 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp"] Jul 19 08:40:25.683592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683564 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0eb9302-643f-4535-999e-258fca42136d" containerName="main" Jul 19 08:40:25.683592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683574 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0eb9302-643f-4535-999e-258fca42136d" containerName="main" Jul 19 08:40:25.683592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683582 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488de101-d58f-485e-8a97-8dd1e66b1cdd" containerName="main" Jul 19 08:40:25.683592 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683589 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="488de101-d58f-485e-8a97-8dd1e66b1cdd" containerName="main" Jul 19 08:40:25.683763 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683648 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0eb9302-643f-4535-999e-258fca42136d" containerName="main" Jul 19 08:40:25.683763 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.683658 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="488de101-d58f-485e-8a97-8dd1e66b1cdd" containerName="main" Jul 19 08:40:25.686580 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.686563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:25.688825 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.688800 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:40:25.693234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.693201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp"] Jul 19 08:40:25.792148 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.792109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcf2ec4-4151-44e2-8257-054bfc4b086e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-brphp\" (UID: \"0dcf2ec4-4151-44e2-8257-054bfc4b086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:25.893086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.893048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcf2ec4-4151-44e2-8257-054bfc4b086e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-brphp\" (UID: \"0dcf2ec4-4151-44e2-8257-054bfc4b086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:25.893741 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.893718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcf2ec4-4151-44e2-8257-054bfc4b086e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-brphp\" (UID: \"0dcf2ec4-4151-44e2-8257-054bfc4b086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:25.998311 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:25.998224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:26.133254 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:26.133219 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp"] Jul 19 08:40:26.136690 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:40:26.136636 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcf2ec4_4151_44e2_8257_054bfc4b086e.slice/crio-60c419921112c16b2f83076c83a5452db219920cff30223d57385ebaa1ca4912 WatchSource:0}: Error finding container 60c419921112c16b2f83076c83a5452db219920cff30223d57385ebaa1ca4912: Status 404 returned error can't find the container with id 60c419921112c16b2f83076c83a5452db219920cff30223d57385ebaa1ca4912 Jul 19 08:40:26.310984 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:26.310958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" event={"ID":"0dcf2ec4-4151-44e2-8257-054bfc4b086e","Type":"ContainerStarted","Data":"60c419921112c16b2f83076c83a5452db219920cff30223d57385ebaa1ca4912"} Jul 19 08:40:27.317567 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:27.317529 2574 generic.go:356] "Generic (PLEG): container finished" podID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerID="0b58d4a84aa01b2a0110c1989614145a42007bc89614e8eb0abeea85f3041de9" exitCode=6 Jul 19 08:40:27.318054 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:27.317616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" event={"ID":"0dcf2ec4-4151-44e2-8257-054bfc4b086e","Type":"ContainerDied","Data":"0b58d4a84aa01b2a0110c1989614145a42007bc89614e8eb0abeea85f3041de9"} Jul 19 08:40:27.318054 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:27.317805 2574 scope.go:117] "RemoveContainer" containerID="0b58d4a84aa01b2a0110c1989614145a42007bc89614e8eb0abeea85f3041de9" Jul 19 08:40:28.325231 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:28.325135 2574 generic.go:356] "Generic (PLEG): container finished" podID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerID="648eff2aa68d589fc5175e435130e9ddaa8eab02451041304d5cc155ba314a3e" exitCode=6 Jul 19 08:40:28.325643 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:28.325234 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" event={"ID":"0dcf2ec4-4151-44e2-8257-054bfc4b086e","Type":"ContainerDied","Data":"648eff2aa68d589fc5175e435130e9ddaa8eab02451041304d5cc155ba314a3e"} Jul 19 08:40:28.325643 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:28.325334 2574 scope.go:117] "RemoveContainer" containerID="0b58d4a84aa01b2a0110c1989614145a42007bc89614e8eb0abeea85f3041de9" Jul 19 08:40:28.325643 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:28.325531 2574 scope.go:117] "RemoveContainer" containerID="648eff2aa68d589fc5175e435130e9ddaa8eab02451041304d5cc155ba314a3e" Jul 19 08:40:28.325813 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:40:28.325790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-brphp_odh-ai-gateway-infra(0dcf2ec4-4151-44e2-8257-054bfc4b086e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" Jul 19 08:40:29.331142 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:29.331113 2574 scope.go:117] "RemoveContainer" containerID="648eff2aa68d589fc5175e435130e9ddaa8eab02451041304d5cc155ba314a3e" Jul 19 08:40:29.331620 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:40:29.331390 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-brphp_odh-ai-gateway-infra(0dcf2ec4-4151-44e2-8257-054bfc4b086e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" Jul 19 08:40:40.726964 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:40.726930 2574 scope.go:117] "RemoveContainer" containerID="648eff2aa68d589fc5175e435130e9ddaa8eab02451041304d5cc155ba314a3e" Jul 19 08:40:41.388765 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:41.388726 2574 generic.go:356] "Generic (PLEG): container finished" podID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerID="400402c971168ea3cf4d256223710f891e5cf2d6a2706a69abfda3227042e236" exitCode=6 Jul 19 08:40:41.388939 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:41.388796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" event={"ID":"0dcf2ec4-4151-44e2-8257-054bfc4b086e","Type":"ContainerDied","Data":"400402c971168ea3cf4d256223710f891e5cf2d6a2706a69abfda3227042e236"} Jul 19 08:40:41.388939 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:41.388890 2574 scope.go:117] "RemoveContainer" containerID="648eff2aa68d589fc5175e435130e9ddaa8eab02451041304d5cc155ba314a3e" Jul 19 08:40:41.389297 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:41.389275 2574 scope.go:117] "RemoveContainer" containerID="400402c971168ea3cf4d256223710f891e5cf2d6a2706a69abfda3227042e236" Jul 19 08:40:41.389560 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:40:41.389537 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-brphp_odh-ai-gateway-infra(0dcf2ec4-4151-44e2-8257-054bfc4b086e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" Jul 19 08:40:42.415604 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:42.415570 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp"] Jul 19 08:40:42.555701 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:42.555654 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:42.662190 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:42.662153 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcf2ec4-4151-44e2-8257-054bfc4b086e-maas-api-service-ca\") pod \"0dcf2ec4-4151-44e2-8257-054bfc4b086e\" (UID: \"0dcf2ec4-4151-44e2-8257-054bfc4b086e\") " Jul 19 08:40:42.662528 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:42.662501 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcf2ec4-4151-44e2-8257-054bfc4b086e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dcf2ec4-4151-44e2-8257-054bfc4b086e" (UID: "0dcf2ec4-4151-44e2-8257-054bfc4b086e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:40:42.763374 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:42.763331 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcf2ec4-4151-44e2-8257-054bfc4b086e-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:43.400342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:43.400302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" event={"ID":"0dcf2ec4-4151-44e2-8257-054bfc4b086e","Type":"ContainerDied","Data":"60c419921112c16b2f83076c83a5452db219920cff30223d57385ebaa1ca4912"} Jul 19 08:40:43.400342 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:43.400339 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp" Jul 19 08:40:43.400587 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:43.400382 2574 scope.go:117] "RemoveContainer" containerID="400402c971168ea3cf4d256223710f891e5cf2d6a2706a69abfda3227042e236" Jul 19 08:40:43.414772 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:43.414745 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp"] Jul 19 08:40:43.419113 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:43.419086 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-brphp"] Jul 19 08:40:44.731761 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:44.731723 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" path="/var/lib/kubelet/pods/0dcf2ec4-4151-44e2-8257-054bfc4b086e/volumes" Jul 19 08:40:48.629343 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:48.629307 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8"] Jul 19 08:40:48.629799 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:48.629579 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" podUID="7312a3cc-99e7-44de-80d9-3c7b046b3727" containerName="maas-api" containerID="cri-o://694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7" gracePeriod=30 Jul 19 08:40:49.080438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.080409 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:40:49.228285 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.228189 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbt2n\" (UniqueName: \"kubernetes.io/projected/7312a3cc-99e7-44de-80d9-3c7b046b3727-kube-api-access-fbt2n\") pod \"7312a3cc-99e7-44de-80d9-3c7b046b3727\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " Jul 19 08:40:49.228285 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.228236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312a3cc-99e7-44de-80d9-3c7b046b3727-openshift-service-ca\") pod \"7312a3cc-99e7-44de-80d9-3c7b046b3727\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " Jul 19 08:40:49.228523 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.228295 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls\") pod \"7312a3cc-99e7-44de-80d9-3c7b046b3727\" (UID: \"7312a3cc-99e7-44de-80d9-3c7b046b3727\") " Jul 19 08:40:49.228642 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.228616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7312a3cc-99e7-44de-80d9-3c7b046b3727-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7312a3cc-99e7-44de-80d9-3c7b046b3727" (UID: "7312a3cc-99e7-44de-80d9-3c7b046b3727"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:40:49.230628 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.230603 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7312a3cc-99e7-44de-80d9-3c7b046b3727-kube-api-access-fbt2n" (OuterVolumeSpecName: "kube-api-access-fbt2n") pod "7312a3cc-99e7-44de-80d9-3c7b046b3727" (UID: "7312a3cc-99e7-44de-80d9-3c7b046b3727"). InnerVolumeSpecName "kube-api-access-fbt2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 08:40:49.230771 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.230629 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7312a3cc-99e7-44de-80d9-3c7b046b3727" (UID: "7312a3cc-99e7-44de-80d9-3c7b046b3727"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 08:40:49.329105 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.329070 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312a3cc-99e7-44de-80d9-3c7b046b3727-maas-api-tls\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:49.329105 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.329103 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbt2n\" (UniqueName: \"kubernetes.io/projected/7312a3cc-99e7-44de-80d9-3c7b046b3727-kube-api-access-fbt2n\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:49.329105 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.329113 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312a3cc-99e7-44de-80d9-3c7b046b3727-openshift-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:40:49.430277 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.430243 2574 generic.go:356] "Generic (PLEG): container finished" podID="7312a3cc-99e7-44de-80d9-3c7b046b3727" containerID="694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7" exitCode=0 Jul 19 08:40:49.430450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.430284 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" Jul 19 08:40:49.430450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.430310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" event={"ID":"7312a3cc-99e7-44de-80d9-3c7b046b3727","Type":"ContainerDied","Data":"694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7"} Jul 19 08:40:49.430450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.430352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8" event={"ID":"7312a3cc-99e7-44de-80d9-3c7b046b3727","Type":"ContainerDied","Data":"13696b56d501a94d1b5664b626707296480f88bba12bf2f3700fb6bfb964f73d"} Jul 19 08:40:49.430450 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.430373 2574 scope.go:117] "RemoveContainer" containerID="694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7" Jul 19 08:40:49.439952 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.439933 2574 scope.go:117] "RemoveContainer" containerID="694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7" Jul 19 08:40:49.440222 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:40:49.440205 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7\": container with ID starting with 694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7 not found: ID does not exist" containerID="694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7" Jul 19 08:40:49.440270 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.440232 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7"} err="failed to get container status \"694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7\": rpc error: code = NotFound desc = could not find container \"694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7\": container with ID starting with 694a2c3235482abc15f507874f02e9dd640f1c2e3484f8405773644f8eb4dda7 not found: ID does not exist" Jul 19 08:40:49.450547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.450522 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8"] Jul 19 08:40:49.452212 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:49.452191 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-190fde-786d85c676-wfrv8"] Jul 19 08:40:50.731640 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:50.731605 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7312a3cc-99e7-44de-80d9-3c7b046b3727" path="/var/lib/kubelet/pods/7312a3cc-99e7-44de-80d9-3c7b046b3727/volumes" Jul 19 08:40:58.026964 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.026881 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs"] Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027234 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7312a3cc-99e7-44de-80d9-3c7b046b3727" containerName="maas-api" Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027244 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7312a3cc-99e7-44de-80d9-3c7b046b3727" containerName="maas-api" Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027255 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027261 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027269 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027275 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.027349 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027345 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7312a3cc-99e7-44de-80d9-3c7b046b3727" containerName="maas-api" Jul 19 08:40:58.027569 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027353 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.027569 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027360 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.027569 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.027367 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:40:58.031634 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.031617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:40:58.034139 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.034114 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:40:58.036176 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.036154 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs"] Jul 19 08:40:58.102752 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.102703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43d9d7a-9696-43ba-af33-6ebfe72d40c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-d64cs\" (UID: \"b43d9d7a-9696-43ba-af33-6ebfe72d40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:40:58.203796 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.203757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43d9d7a-9696-43ba-af33-6ebfe72d40c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-d64cs\" (UID: \"b43d9d7a-9696-43ba-af33-6ebfe72d40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:40:58.204387 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.204368 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43d9d7a-9696-43ba-af33-6ebfe72d40c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-d64cs\" (UID: \"b43d9d7a-9696-43ba-af33-6ebfe72d40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:40:58.343477 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.343446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:40:58.475179 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:58.475058 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs"] Jul 19 08:40:58.477388 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:40:58.477354 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43d9d7a_9696_43ba_af33_6ebfe72d40c5.slice/crio-b72a3ca37086c024560c184695d263b477608521373a25bdf5a3e2af1aed075c WatchSource:0}: Error finding container b72a3ca37086c024560c184695d263b477608521373a25bdf5a3e2af1aed075c: Status 404 returned error can't find the container with id b72a3ca37086c024560c184695d263b477608521373a25bdf5a3e2af1aed075c Jul 19 08:40:59.482731 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:59.482606 2574 generic.go:356] "Generic (PLEG): container finished" podID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerID="ac0161956919fa49b8a33cc2fcef57b517a189cb3aaaa9808afe542bf9c90de8" exitCode=6 Jul 19 08:40:59.482731 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:59.482714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" event={"ID":"b43d9d7a-9696-43ba-af33-6ebfe72d40c5","Type":"ContainerDied","Data":"ac0161956919fa49b8a33cc2fcef57b517a189cb3aaaa9808afe542bf9c90de8"} Jul 19 08:40:59.483273 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:59.482761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" event={"ID":"b43d9d7a-9696-43ba-af33-6ebfe72d40c5","Type":"ContainerStarted","Data":"b72a3ca37086c024560c184695d263b477608521373a25bdf5a3e2af1aed075c"} Jul 19 08:40:59.483273 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:40:59.482964 2574 scope.go:117] "RemoveContainer" containerID="ac0161956919fa49b8a33cc2fcef57b517a189cb3aaaa9808afe542bf9c90de8" Jul 19 08:41:00.488486 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.488453 2574 generic.go:356] "Generic (PLEG): container finished" podID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerID="5e8c60d301dbcc61330e5a59dffd2b09a9cdb750b2000f7a3efcb2081ce59fef" exitCode=6 Jul 19 08:41:00.488909 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.488551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" event={"ID":"b43d9d7a-9696-43ba-af33-6ebfe72d40c5","Type":"ContainerDied","Data":"5e8c60d301dbcc61330e5a59dffd2b09a9cdb750b2000f7a3efcb2081ce59fef"} Jul 19 08:41:00.488909 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.488652 2574 scope.go:117] "RemoveContainer" containerID="ac0161956919fa49b8a33cc2fcef57b517a189cb3aaaa9808afe542bf9c90de8" Jul 19 08:41:00.488909 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.488746 2574 scope.go:117] "RemoveContainer" containerID="5e8c60d301dbcc61330e5a59dffd2b09a9cdb750b2000f7a3efcb2081ce59fef" Jul 19 08:41:00.489052 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:00.488975 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-d64cs_odh-ai-gateway-infra(b43d9d7a-9696-43ba-af33-6ebfe72d40c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" Jul 19 08:41:00.780417 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.780333 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt"] Jul 19 08:41:00.780725 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.780713 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:41:00.780782 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.780726 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcf2ec4-4151-44e2-8257-054bfc4b086e" containerName="revoke-keys" Jul 19 08:41:00.785110 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.785093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:00.790237 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.790207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt"] Jul 19 08:41:00.829014 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.828975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23154326-d82d-460d-aa1e-39d0ac43df71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt\" (UID: \"23154326-d82d-460d-aa1e-39d0ac43df71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:00.929670 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.929635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23154326-d82d-460d-aa1e-39d0ac43df71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt\" (UID: \"23154326-d82d-460d-aa1e-39d0ac43df71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:00.930331 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:00.930308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23154326-d82d-460d-aa1e-39d0ac43df71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt\" (UID: \"23154326-d82d-460d-aa1e-39d0ac43df71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:01.096633 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:01.096601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:01.231354 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:01.231322 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt"] Jul 19 08:41:01.234170 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:41:01.234135 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23154326_d82d_460d_aa1e_39d0ac43df71.slice/crio-43c8eca62dfcb500b662bcfa9ba0727c80083899b0b967fcc96a3c8b945510e6 WatchSource:0}: Error finding container 43c8eca62dfcb500b662bcfa9ba0727c80083899b0b967fcc96a3c8b945510e6: Status 404 returned error can't find the container with id 43c8eca62dfcb500b662bcfa9ba0727c80083899b0b967fcc96a3c8b945510e6 Jul 19 08:41:01.495768 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:01.495740 2574 scope.go:117] "RemoveContainer" containerID="5e8c60d301dbcc61330e5a59dffd2b09a9cdb750b2000f7a3efcb2081ce59fef" Jul 19 08:41:01.496155 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:01.495990 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-d64cs_odh-ai-gateway-infra(b43d9d7a-9696-43ba-af33-6ebfe72d40c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" Jul 19 08:41:01.497009 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:01.496982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" event={"ID":"23154326-d82d-460d-aa1e-39d0ac43df71","Type":"ContainerStarted","Data":"43c8eca62dfcb500b662bcfa9ba0727c80083899b0b967fcc96a3c8b945510e6"} Jul 19 08:41:01.524214 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:01.524159 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" podStartSLOduration=1.524145065 podStartE2EDuration="1.524145065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:01.521669372 +0000 UTC m=+3509.301332686" watchObservedRunningTime="2026-07-19 08:41:01.524145065 +0000 UTC m=+3509.303808378" Jul 19 08:41:02.503113 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:02.503079 2574 generic.go:356] "Generic (PLEG): container finished" podID="23154326-d82d-460d-aa1e-39d0ac43df71" containerID="5aa2cd8f348b04416ffed1992f9e707288b6191a672fa6b02cb193f187e61eb7" exitCode=6 Jul 19 08:41:02.503538 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:02.503165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" event={"ID":"23154326-d82d-460d-aa1e-39d0ac43df71","Type":"ContainerDied","Data":"5aa2cd8f348b04416ffed1992f9e707288b6191a672fa6b02cb193f187e61eb7"} Jul 19 08:41:02.503538 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:02.503341 2574 scope.go:117] "RemoveContainer" containerID="5aa2cd8f348b04416ffed1992f9e707288b6191a672fa6b02cb193f187e61eb7" Jul 19 08:41:03.509887 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:03.509797 2574 generic.go:356] "Generic (PLEG): container finished" podID="23154326-d82d-460d-aa1e-39d0ac43df71" containerID="36f3fc93252d8e905e9d841c18bb6d12683a144f1c7cde88d08e9e62bb673231" exitCode=6 Jul 19 08:41:03.509887 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:03.509880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" event={"ID":"23154326-d82d-460d-aa1e-39d0ac43df71","Type":"ContainerDied","Data":"36f3fc93252d8e905e9d841c18bb6d12683a144f1c7cde88d08e9e62bb673231"} Jul 19 08:41:03.510350 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:03.509932 2574 scope.go:117] "RemoveContainer" containerID="5aa2cd8f348b04416ffed1992f9e707288b6191a672fa6b02cb193f187e61eb7" Jul 19 08:41:03.510350 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:03.510111 2574 scope.go:117] "RemoveContainer" containerID="36f3fc93252d8e905e9d841c18bb6d12683a144f1c7cde88d08e9e62bb673231" Jul 19 08:41:03.510418 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:03.510367 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt_odh-ai-gateway-infra(23154326-d82d-460d-aa1e-39d0ac43df71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" Jul 19 08:41:04.516055 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:04.516028 2574 scope.go:117] "RemoveContainer" containerID="36f3fc93252d8e905e9d841c18bb6d12683a144f1c7cde88d08e9e62bb673231" Jul 19 08:41:04.516443 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:04.516227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt_odh-ai-gateway-infra(23154326-d82d-460d-aa1e-39d0ac43df71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" Jul 19 08:41:09.769188 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:09.769158 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kgr97_c471b15a-d8f5-4d24-9dbf-f46b7e5f0038/manager/0.log" Jul 19 08:41:10.009086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:10.009051 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58f4f5887c-8xcpd_6b1b6506-4b16-4b3a-98bb-45d6d616874e/manager/0.log" Jul 19 08:41:10.326234 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:10.326208 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-2wlfq_cff2e13b-0644-4cc6-be23-f49ed171dde0/copy-manifests/0.log" Jul 19 08:41:10.332621 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:10.332585 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-2wlfq_cff2e13b-0644-4cc6-be23-f49ed171dde0/manager/0.log" Jul 19 08:41:10.941077 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:10.941049 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8694597d9b-n8b7c_95dcbaf9-aedd-4331-a0ef-0f247158d051/maas-api/0.log" Jul 19 08:41:11.271648 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:11.271561 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29740830-xjv99_7e09ef39-3cef-4407-b9b5-fd5db4ddd710/cleanup/0.log" Jul 19 08:41:11.381976 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:11.381941 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-ef124b28-d64cs_b43d9d7a-9696-43ba-af33-6ebfe72d40c5/revoke-keys/1.log" Jul 19 08:41:11.601850 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:11.601824 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-2d2deb-jftbx_8b32af2f-9bd2-4e4d-8905-d2fa8d23e0c2/revoke-keys/0.log" Jul 19 08:41:11.714495 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:11.714465 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt_23154326-d82d-460d-aa1e-39d0ac43df71/revoke-keys/1.log" Jul 19 08:41:12.139895 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:12.139867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-6eff15b1-lmtdk_f1d8cc50-b19c-4652-83d4-fad0092a19ad/revoke-keys/0.log" Jul 19 08:41:12.458045 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:12.457955 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-49dbs_518d77d6-702e-4aca-ae86-fd2fae906ab6/postgres/0.log" Jul 19 08:41:13.992466 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:13.992437 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4w9rt_63d8ba89-d8a8-4285-abcd-05672cfa1d8f/kuadrant-console-plugin/0.log" Jul 19 08:41:14.032137 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.032103 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6"] Jul 19 08:41:14.037011 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.036994 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:14.041663 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.041632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6"] Jul 19 08:41:14.156483 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.156438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13932672-bf4b-49bf-b588-fca7418ee951-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6\" (UID: \"13932672-bf4b-49bf-b588-fca7418ee951\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:14.257347 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.257248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13932672-bf4b-49bf-b588-fca7418ee951-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6\" (UID: \"13932672-bf4b-49bf-b588-fca7418ee951\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:14.257918 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.257900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13932672-bf4b-49bf-b588-fca7418ee951-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6\" (UID: \"13932672-bf4b-49bf-b588-fca7418ee951\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:14.350477 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.350440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:14.489096 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.489066 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6"] Jul 19 08:41:14.491266 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:41:14.491238 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13932672_bf4b_49bf_b588_fca7418ee951.slice/crio-8cd7283adb40896b2d813dd1809ba976381feca1cab4f75a82615b9c71430281 WatchSource:0}: Error finding container 8cd7283adb40896b2d813dd1809ba976381feca1cab4f75a82615b9c71430281: Status 404 returned error can't find the container with id 8cd7283adb40896b2d813dd1809ba976381feca1cab4f75a82615b9c71430281 Jul 19 08:41:14.562846 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.562811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" event={"ID":"13932672-bf4b-49bf-b588-fca7418ee951","Type":"ContainerStarted","Data":"8cd7283adb40896b2d813dd1809ba976381feca1cab4f75a82615b9c71430281"} Jul 19 08:41:14.987850 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:14.987815 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-6h6nz_aac1fabb-3259-4c8e-8dfd-7a3d462cec9a/kube-auth-proxy/0.log" Jul 19 08:41:15.448368 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:15.448334 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-q7f2k_4550ca18-d965-4ccb-a38e-c46d0dc90aaa/payload-processing/0.log" Jul 19 08:41:15.569079 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:15.569042 2574 generic.go:356] "Generic (PLEG): container finished" podID="13932672-bf4b-49bf-b588-fca7418ee951" containerID="779f2c0140c1b589d694385057f038ca9028a3f0cbdeb5e0891a78e60f1a355c" exitCode=6 Jul 19 08:41:15.569267 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:15.569100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" event={"ID":"13932672-bf4b-49bf-b588-fca7418ee951","Type":"ContainerDied","Data":"779f2c0140c1b589d694385057f038ca9028a3f0cbdeb5e0891a78e60f1a355c"} Jul 19 08:41:15.569378 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:15.569365 2574 scope.go:117] "RemoveContainer" containerID="779f2c0140c1b589d694385057f038ca9028a3f0cbdeb5e0891a78e60f1a355c" Jul 19 08:41:15.726262 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:15.726177 2574 scope.go:117] "RemoveContainer" containerID="5e8c60d301dbcc61330e5a59dffd2b09a9cdb750b2000f7a3efcb2081ce59fef" Jul 19 08:41:15.887855 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:15.887825 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-p286k_405e3ac0-4aa0-4242-a298-8959224e5eed/main/0.log" Jul 19 08:41:16.099813 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.099785 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wx64b_a508c8d7-6e00-4236-9aad-b334d0125402/main/0.log" Jul 19 08:41:16.419765 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.419731 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-z9npz_68051fdf-91d0-4209-abbf-a7fb5ede2999/main/0.log" Jul 19 08:41:16.576279 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.576192 2574 generic.go:356] "Generic (PLEG): container finished" podID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerID="803989b767464290b793753451915b912379521b8845dd5591ffadb7dae4c12f" exitCode=6 Jul 19 08:41:16.576279 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.576269 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" event={"ID":"b43d9d7a-9696-43ba-af33-6ebfe72d40c5","Type":"ContainerDied","Data":"803989b767464290b793753451915b912379521b8845dd5591ffadb7dae4c12f"} Jul 19 08:41:16.576852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.576325 2574 scope.go:117] "RemoveContainer" containerID="5e8c60d301dbcc61330e5a59dffd2b09a9cdb750b2000f7a3efcb2081ce59fef" Jul 19 08:41:16.576852 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.576643 2574 scope.go:117] "RemoveContainer" containerID="803989b767464290b793753451915b912379521b8845dd5591ffadb7dae4c12f" Jul 19 08:41:16.576956 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:16.576908 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-d64cs_odh-ai-gateway-infra(b43d9d7a-9696-43ba-af33-6ebfe72d40c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" Jul 19 08:41:16.578242 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.578220 2574 generic.go:356] "Generic (PLEG): container finished" podID="13932672-bf4b-49bf-b588-fca7418ee951" containerID="bfb1d1080bb3febe50a6f3ac5fc8c03b63bb9e2409596113597819bd582973ea" exitCode=6 Jul 19 08:41:16.578351 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.578261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" event={"ID":"13932672-bf4b-49bf-b588-fca7418ee951","Type":"ContainerDied","Data":"bfb1d1080bb3febe50a6f3ac5fc8c03b63bb9e2409596113597819bd582973ea"} Jul 19 08:41:16.578439 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.578426 2574 scope.go:117] "RemoveContainer" containerID="bfb1d1080bb3febe50a6f3ac5fc8c03b63bb9e2409596113597819bd582973ea" Jul 19 08:41:16.578627 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:16.578611 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6_odh-ai-gateway-infra(13932672-bf4b-49bf-b588-fca7418ee951)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" podUID="13932672-bf4b-49bf-b588-fca7418ee951" Jul 19 08:41:16.587393 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.587374 2574 scope.go:117] "RemoveContainer" containerID="779f2c0140c1b589d694385057f038ca9028a3f0cbdeb5e0891a78e60f1a355c" Jul 19 08:41:16.726939 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:16.726906 2574 scope.go:117] "RemoveContainer" containerID="36f3fc93252d8e905e9d841c18bb6d12683a144f1c7cde88d08e9e62bb673231" Jul 19 08:41:17.586512 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.586473 2574 generic.go:356] "Generic (PLEG): container finished" podID="23154326-d82d-460d-aa1e-39d0ac43df71" containerID="2fab4f8e3232f42be082a048a575104980647166adcac36ed3e870b925433c33" exitCode=6 Jul 19 08:41:17.586986 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.586550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" event={"ID":"23154326-d82d-460d-aa1e-39d0ac43df71","Type":"ContainerDied","Data":"2fab4f8e3232f42be082a048a575104980647166adcac36ed3e870b925433c33"} Jul 19 08:41:17.586986 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.586614 2574 scope.go:117] "RemoveContainer" containerID="36f3fc93252d8e905e9d841c18bb6d12683a144f1c7cde88d08e9e62bb673231" Jul 19 08:41:17.587124 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.586985 2574 scope.go:117] "RemoveContainer" containerID="2fab4f8e3232f42be082a048a575104980647166adcac36ed3e870b925433c33" Jul 19 08:41:17.587251 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:17.587229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt_odh-ai-gateway-infra(23154326-d82d-460d-aa1e-39d0ac43df71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" Jul 19 08:41:17.588378 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.588362 2574 scope.go:117] "RemoveContainer" containerID="bfb1d1080bb3febe50a6f3ac5fc8c03b63bb9e2409596113597819bd582973ea" Jul 19 08:41:17.588586 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:17.588567 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6_odh-ai-gateway-infra(13932672-bf4b-49bf-b588-fca7418ee951)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" podUID="13932672-bf4b-49bf-b588-fca7418ee951" Jul 19 08:41:17.600080 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.600056 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs"] Jul 19 08:41:17.744086 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.744062 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:41:17.898441 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.898402 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43d9d7a-9696-43ba-af33-6ebfe72d40c5-maas-api-service-ca\") pod \"b43d9d7a-9696-43ba-af33-6ebfe72d40c5\" (UID: \"b43d9d7a-9696-43ba-af33-6ebfe72d40c5\") " Jul 19 08:41:17.898847 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.898826 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43d9d7a-9696-43ba-af33-6ebfe72d40c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b43d9d7a-9696-43ba-af33-6ebfe72d40c5" (UID: "b43d9d7a-9696-43ba-af33-6ebfe72d40c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:41:17.999901 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:17.999815 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43d9d7a-9696-43ba-af33-6ebfe72d40c5-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:41:18.596361 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.596326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" event={"ID":"b43d9d7a-9696-43ba-af33-6ebfe72d40c5","Type":"ContainerDied","Data":"b72a3ca37086c024560c184695d263b477608521373a25bdf5a3e2af1aed075c"} Jul 19 08:41:18.596837 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.596376 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs" Jul 19 08:41:18.596837 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.596417 2574 scope.go:117] "RemoveContainer" containerID="803989b767464290b793753451915b912379521b8845dd5591ffadb7dae4c12f" Jul 19 08:41:18.615208 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.615186 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt"] Jul 19 08:41:18.622692 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.622655 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs"] Jul 19 08:41:18.628314 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.628290 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-d64cs"] Jul 19 08:41:18.733749 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.733714 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" path="/var/lib/kubelet/pods/b43d9d7a-9696-43ba-af33-6ebfe72d40c5/volumes" Jul 19 08:41:18.751030 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.751004 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:18.908128 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.908028 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23154326-d82d-460d-aa1e-39d0ac43df71-maas-api-service-ca\") pod \"23154326-d82d-460d-aa1e-39d0ac43df71\" (UID: \"23154326-d82d-460d-aa1e-39d0ac43df71\") " Jul 19 08:41:18.908400 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:18.908372 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23154326-d82d-460d-aa1e-39d0ac43df71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23154326-d82d-460d-aa1e-39d0ac43df71" (UID: "23154326-d82d-460d-aa1e-39d0ac43df71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:41:19.008854 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:19.008816 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23154326-d82d-460d-aa1e-39d0ac43df71-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:41:19.603386 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:19.603356 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" Jul 19 08:41:19.603386 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:19.603373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt" event={"ID":"23154326-d82d-460d-aa1e-39d0ac43df71","Type":"ContainerDied","Data":"43c8eca62dfcb500b662bcfa9ba0727c80083899b0b967fcc96a3c8b945510e6"} Jul 19 08:41:19.603880 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:19.603438 2574 scope.go:117] "RemoveContainer" containerID="2fab4f8e3232f42be082a048a575104980647166adcac36ed3e870b925433c33" Jul 19 08:41:19.623119 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:19.623092 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt"] Jul 19 08:41:19.628438 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:19.628413 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-4ldxt"] Jul 19 08:41:20.731784 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:20.731751 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" path="/var/lib/kubelet/pods/23154326-d82d-460d-aa1e-39d0ac43df71/volumes" Jul 19 08:41:24.260440 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:24.260401 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vkbl7_8bff89a7-ac8c-48fe-badc-c78dd9f68166/global-pull-secret-syncer/0.log" Jul 19 08:41:24.348465 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:24.348436 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q6lgs_870abfa2-ba6e-4375-b66f-1a2a309118fb/konnectivity-agent/0.log" Jul 19 08:41:24.422755 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:24.422722 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-126.ec2.internal_a13e305d78555894578d5848a397f831/haproxy/0.log" Jul 19 08:41:30.384818 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:30.384792 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4w9rt_63d8ba89-d8a8-4285-abcd-05672cfa1d8f/kuadrant-console-plugin/0.log" Jul 19 08:41:30.726146 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:30.726051 2574 scope.go:117] "RemoveContainer" containerID="bfb1d1080bb3febe50a6f3ac5fc8c03b63bb9e2409596113597819bd582973ea" Jul 19 08:41:31.662864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:31.662765 2574 generic.go:356] "Generic (PLEG): container finished" podID="13932672-bf4b-49bf-b588-fca7418ee951" containerID="458590a7e78d3d306f159be2e05cde8cdaaa5f706a714fddf7fd0c80cae32a75" exitCode=6 Jul 19 08:41:31.662864 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:31.662844 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" event={"ID":"13932672-bf4b-49bf-b588-fca7418ee951","Type":"ContainerDied","Data":"458590a7e78d3d306f159be2e05cde8cdaaa5f706a714fddf7fd0c80cae32a75"} Jul 19 08:41:31.663382 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:31.662954 2574 scope.go:117] "RemoveContainer" containerID="bfb1d1080bb3febe50a6f3ac5fc8c03b63bb9e2409596113597819bd582973ea" Jul 19 08:41:31.663382 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:31.663278 2574 scope.go:117] "RemoveContainer" containerID="458590a7e78d3d306f159be2e05cde8cdaaa5f706a714fddf7fd0c80cae32a75" Jul 19 08:41:31.663497 ip-10-0-132-126 kubenswrapper[2574]: E0719 08:41:31.663481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6_odh-ai-gateway-infra(13932672-bf4b-49bf-b588-fca7418ee951)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" podUID="13932672-bf4b-49bf-b588-fca7418ee951" Jul 19 08:41:32.447822 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.447794 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/alertmanager/0.log" Jul 19 08:41:32.475181 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.475089 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/config-reloader/0.log" Jul 19 08:41:32.497601 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.497573 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/kube-rbac-proxy-web/0.log" Jul 19 08:41:32.518046 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.518021 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/kube-rbac-proxy/0.log" Jul 19 08:41:32.540098 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.540064 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/kube-rbac-proxy-metric/0.log" Jul 19 08:41:32.557992 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.557968 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/prom-label-proxy/0.log" Jul 19 08:41:32.580380 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.580358 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0b2ded24-eb51-4f3c-bb13-1da328f80cd3/init-config-reloader/0.log" Jul 19 08:41:32.687432 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.687394 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6"] Jul 19 08:41:32.748142 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.748118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b4fd4859c-f5nlj_37776d1e-46e0-45f6-9b4d-b5853a7b26ec/metrics-server/0.log" Jul 19 08:41:32.801445 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.801388 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-brp9z_b2b337a1-f938-4bb2-97af-76675f05a4f5/node-exporter/0.log" Jul 19 08:41:32.822082 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.822049 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-brp9z_b2b337a1-f938-4bb2-97af-76675f05a4f5/kube-rbac-proxy/0.log" Jul 19 08:41:32.826645 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.826622 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:32.841662 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.841632 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-brp9z_b2b337a1-f938-4bb2-97af-76675f05a4f5/init-textfile/0.log" Jul 19 08:41:32.941244 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.941210 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13932672-bf4b-49bf-b588-fca7418ee951-maas-api-service-ca\") pod \"13932672-bf4b-49bf-b588-fca7418ee951\" (UID: \"13932672-bf4b-49bf-b588-fca7418ee951\") " Jul 19 08:41:32.941591 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:32.941565 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13932672-bf4b-49bf-b588-fca7418ee951-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13932672-bf4b-49bf-b588-fca7418ee951" (UID: "13932672-bf4b-49bf-b588-fca7418ee951"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 08:41:33.042401 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.042371 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13932672-bf4b-49bf-b588-fca7418ee951-maas-api-service-ca\") on node \"ip-10-0-132-126.ec2.internal\" DevicePath \"\"" Jul 19 08:41:33.114990 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.114952 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/prometheus/0.log" Jul 19 08:41:33.134238 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.134211 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/config-reloader/0.log" Jul 19 08:41:33.153263 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.153236 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/thanos-sidecar/0.log" Jul 19 08:41:33.171867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.171820 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/kube-rbac-proxy-web/0.log" Jul 19 08:41:33.190321 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.190296 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/kube-rbac-proxy/0.log" Jul 19 08:41:33.208428 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.208399 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/kube-rbac-proxy-thanos/0.log" Jul 19 08:41:33.225873 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.225849 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fae95004-cea2-4710-a5b6-d3bb5c60b5dc/init-config-reloader/0.log" Jul 19 08:41:33.387357 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.387330 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b668f6f5-2jwcq_48f3dac1-e06b-48b0-ad27-c069c3c41cd4/thanos-query/0.log" Jul 19 08:41:33.410552 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.410527 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b668f6f5-2jwcq_48f3dac1-e06b-48b0-ad27-c069c3c41cd4/kube-rbac-proxy-web/0.log" Jul 19 08:41:33.430156 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.430122 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b668f6f5-2jwcq_48f3dac1-e06b-48b0-ad27-c069c3c41cd4/kube-rbac-proxy/0.log" Jul 19 08:41:33.447411 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.447384 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b668f6f5-2jwcq_48f3dac1-e06b-48b0-ad27-c069c3c41cd4/prom-label-proxy/0.log" Jul 19 08:41:33.465733 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.465707 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b668f6f5-2jwcq_48f3dac1-e06b-48b0-ad27-c069c3c41cd4/kube-rbac-proxy-rules/0.log" Jul 19 08:41:33.483878 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.483848 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b668f6f5-2jwcq_48f3dac1-e06b-48b0-ad27-c069c3c41cd4/kube-rbac-proxy-metrics/0.log" Jul 19 08:41:33.675662 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.675569 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" event={"ID":"13932672-bf4b-49bf-b588-fca7418ee951","Type":"ContainerDied","Data":"8cd7283adb40896b2d813dd1809ba976381feca1cab4f75a82615b9c71430281"} Jul 19 08:41:33.675662 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.675599 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6" Jul 19 08:41:33.675662 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.675625 2574 scope.go:117] "RemoveContainer" containerID="458590a7e78d3d306f159be2e05cde8cdaaa5f706a714fddf7fd0c80cae32a75" Jul 19 08:41:33.701853 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.701819 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6"] Jul 19 08:41:33.704805 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:33.704782 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54gm6"] Jul 19 08:41:34.456401 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456364 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9"] Jul 19 08:41:34.456806 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456791 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456808 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456818 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456824 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456831 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456836 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456843 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456848 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456854 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456859 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.456875 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456877 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456882 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456939 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456947 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456953 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456959 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456965 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456970 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:34.457183 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.456976 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:34.461780 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.461758 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.464032 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.464011 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nlzs\"/\"kube-root-ca.crt\"" Jul 19 08:41:34.464337 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.464308 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nlzs\"/\"openshift-service-ca.crt\"" Jul 19 08:41:34.464837 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.464816 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nlzs\"/\"default-dockercfg-5jrnw\"" Jul 19 08:41:34.467415 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.467390 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9"] Jul 19 08:41:34.552366 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.552330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-lib-modules\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.552366 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.552367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5rv\" (UniqueName: \"kubernetes.io/projected/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-kube-api-access-rn5rv\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.552614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.552396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-podres\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.552614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.552476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-sys\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.552614 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.552527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-proc\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653078 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-lib-modules\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653078 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5rv\" (UniqueName: \"kubernetes.io/projected/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-kube-api-access-rn5rv\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653318 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-podres\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653318 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-lib-modules\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653318 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-sys\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653318 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-proc\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653318 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-podres\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653540 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653350 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-proc\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.653540 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.653401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-sys\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.660821 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.660794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5rv\" (UniqueName: \"kubernetes.io/projected/fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7-kube-api-access-rn5rv\") pod \"perf-node-gather-daemonset-26st9\" (UID: \"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.732453 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.732366 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13932672-bf4b-49bf-b588-fca7418ee951" path="/var/lib/kubelet/pods/13932672-bf4b-49bf-b588-fca7418ee951/volumes" Jul 19 08:41:34.773211 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.773183 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:34.906413 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:34.906371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9"] Jul 19 08:41:34.908737 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:41:34.908709 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa06e29e_c3c9_4e18_b7f4_eb31f5de53d7.slice/crio-37e4c5704cecdf6eef576160c77f12e864406b7de46e120946a0297ac711b7a0 WatchSource:0}: Error finding container 37e4c5704cecdf6eef576160c77f12e864406b7de46e120946a0297ac711b7a0: Status 404 returned error can't find the container with id 37e4c5704cecdf6eef576160c77f12e864406b7de46e120946a0297ac711b7a0 Jul 19 08:41:35.688759 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:35.688717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" event={"ID":"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7","Type":"ContainerStarted","Data":"fc8ff635ef79968f1fa32a048acf28ca3ae0b42fe132961070c09d4e339ba137"} Jul 19 08:41:35.688936 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:35.688779 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" event={"ID":"fa06e29e-c3c9-4e18-b7f4-eb31f5de53d7","Type":"ContainerStarted","Data":"37e4c5704cecdf6eef576160c77f12e864406b7de46e120946a0297ac711b7a0"} Jul 19 08:41:35.688936 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:35.688838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:35.705294 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:35.705243 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" podStartSLOduration=1.705228833 podStartE2EDuration="1.705228833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 08:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:35.703785884 +0000 UTC m=+3543.483449222" watchObservedRunningTime="2026-07-19 08:41:35.705228833 +0000 UTC m=+3543.484892146" Jul 19 08:41:37.256379 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:37.256355 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7tll8_e52bb4ff-66ba-413d-94e7-1a2988168d63/dns/0.log" Jul 19 08:41:37.274745 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:37.274720 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7tll8_e52bb4ff-66ba-413d-94e7-1a2988168d63/kube-rbac-proxy/0.log" Jul 19 08:41:37.428083 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:37.428054 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qnz7d_036351e7-17ab-4846-8d06-e67caf447ba0/dns-node-resolver/0.log" Jul 19 08:41:37.945920 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:37.945880 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-845bbc4585-824dm_bd124b76-fc39-4a0b-9ada-f6f9aa15e8c8/registry/0.log" Jul 19 08:41:37.988174 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:37.988132 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d59n6_41e5e5d2-41f9-43b6-b303-ef986ddb5a26/node-ca/0.log" Jul 19 08:41:39.016861 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:39.016826 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-6h6nz_aac1fabb-3259-4c8e-8dfd-7a3d462cec9a/kube-auth-proxy/0.log" Jul 19 08:41:39.164947 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:39.164906 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-q7f2k_4550ca18-d965-4ccb-a38e-c46d0dc90aaa/payload-processing/0.log" Jul 19 08:41:39.848214 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:39.848188 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j6frs_c4539355-46df-450c-bbe7-38024d9068a1/serve-healthcheck-canary/0.log" Jul 19 08:41:40.391504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:40.391477 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d75m9_02dfe2e2-f207-4040-9857-1b45b5267e07/kube-rbac-proxy/0.log" Jul 19 08:41:40.410560 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:40.410537 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d75m9_02dfe2e2-f207-4040-9857-1b45b5267e07/exporter/0.log" Jul 19 08:41:40.439012 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:40.438978 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d75m9_02dfe2e2-f207-4040-9857-1b45b5267e07/extractor/0.log" Jul 19 08:41:41.703446 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:41.703416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-26st9" Jul 19 08:41:42.693338 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:42.693309 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kgr97_c471b15a-d8f5-4d24-9dbf-f46b7e5f0038/manager/0.log" Jul 19 08:41:43.081945 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:43.081908 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58f4f5887c-8xcpd_6b1b6506-4b16-4b3a-98bb-45d6d616874e/manager/0.log" Jul 19 08:41:43.211867 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:43.211821 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-2wlfq_cff2e13b-0644-4cc6-be23-f49ed171dde0/manager/0.log" Jul 19 08:41:43.229902 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:43.229876 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-2wlfq_cff2e13b-0644-4cc6-be23-f49ed171dde0/copy-manifests/0.log" Jul 19 08:41:44.787846 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:44.787817 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-747d9ff956-lchls_af121060-cd61-494e-8b87-13ba67a89698/manager/0.log" Jul 19 08:41:45.525780 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:45.525738 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-6h6nz_aac1fabb-3259-4c8e-8dfd-7a3d462cec9a/kube-auth-proxy/0.log" Jul 19 08:41:45.641286 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:45.641247 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-q7f2k_4550ca18-d965-4ccb-a38e-c46d0dc90aaa/payload-processing/0.log" Jul 19 08:41:46.963761 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:46.963734 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-56f59f876d-6h6nz_aac1fabb-3259-4c8e-8dfd-7a3d462cec9a/kube-auth-proxy/0.log" Jul 19 08:41:47.080547 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:47.080514 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-q7f2k_4550ca18-d965-4ccb-a38e-c46d0dc90aaa/payload-processing/0.log" Jul 19 08:41:50.822145 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822104 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw"] Jul 19 08:41:50.822544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822483 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:50.822544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822494 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:50.822544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822503 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:50.822544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822509 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:50.822544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822516 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:50.822544 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822521 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="13932672-bf4b-49bf-b588-fca7418ee951" containerName="revoke-keys" Jul 19 08:41:50.822777 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822573 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43d9d7a-9696-43ba-af33-6ebfe72d40c5" containerName="revoke-keys" Jul 19 08:41:50.822777 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.822584 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="23154326-d82d-460d-aa1e-39d0ac43df71" containerName="revoke-keys" Jul 19 08:41:50.831693 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.831645 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw"] Jul 19 08:41:50.831926 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.831803 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" Jul 19 08:41:50.834504 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.834481 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\"" Jul 19 08:41:50.884020 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.883972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f3564-c856-4d59-b643-530e9e23a7c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw\" (UID: \"9b5f3564-c856-4d59-b643-530e9e23a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" Jul 19 08:41:50.984549 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.984518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f3564-c856-4d59-b643-530e9e23a7c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw\" (UID: \"9b5f3564-c856-4d59-b643-530e9e23a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" Jul 19 08:41:50.985325 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:50.985298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5f3564-c856-4d59-b643-530e9e23a7c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw\" (UID: \"9b5f3564-c856-4d59-b643-530e9e23a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" Jul 19 08:41:51.015067 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.015032 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6h2ps_3578e380-7d65-4015-a9a0-acf3ee6b5456/kube-multus/0.log" Jul 19 08:41:51.142522 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.142488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" Jul 19 08:41:51.279935 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.279907 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw"] Jul 19 08:41:51.282241 ip-10-0-132-126 kubenswrapper[2574]: W0719 08:41:51.282215 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5f3564_c856_4d59_b643_530e9e23a7c6.slice/crio-93d0f4b5cc7d5ad0b8897a59999a5cd5a2d1de6071101de6b37ff4303416b8bd WatchSource:0}: Error finding container 93d0f4b5cc7d5ad0b8897a59999a5cd5a2d1de6071101de6b37ff4303416b8bd: Status 404 returned error can't find the container with id 93d0f4b5cc7d5ad0b8897a59999a5cd5a2d1de6071101de6b37ff4303416b8bd Jul 19 08:41:51.356436 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.356411 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/kube-multus-additional-cni-plugins/0.log" Jul 19 08:41:51.374567 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.374542 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/egress-router-binary-copy/0.log" Jul 19 08:41:51.398129 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.398044 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/cni-plugins/0.log" Jul 19 08:41:51.416471 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.416449 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/bond-cni-plugin/0.log" Jul 19 08:41:51.434320 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.434291 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/routeoverride-cni/0.log" Jul 19 08:41:51.452999 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.452970 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/whereabouts-cni-bincopy/0.log" Jul 19 08:41:51.470687 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.470641 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qb24p_e9cc735d-0114-4c55-839c-574f1c90d2bf/whereabouts-cni/0.log" Jul 19 08:41:51.573525 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.573494 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4znm8_f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c/network-metrics-daemon/0.log" Jul 19 08:41:51.586985 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.586962 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4znm8_f34c1ae7-5ff5-4f73-8c5c-8ecf0078930c/kube-rbac-proxy/0.log" Jul 19 08:41:51.764249 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.764161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" event={"ID":"9b5f3564-c856-4d59-b643-530e9e23a7c6","Type":"ContainerStarted","Data":"a9082f7ee290d349e9b788871f620880e261f4b0fda30a9ac52c05b485e70840"} Jul 19 08:41:51.764249 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.764231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" event={"ID":"9b5f3564-c856-4d59-b643-530e9e23a7c6","Type":"ContainerStarted","Data":"93d0f4b5cc7d5ad0b8897a59999a5cd5a2d1de6071101de6b37ff4303416b8bd"} Jul 19 08:41:51.778505 ip-10-0-132-126 kubenswrapper[2574]: I0719 08:41:51.778451 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-zlzfw" podStartSLOduration=1.778435555 podStartE2EDuration="1.778435555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:51.777326902 +0000 UTC m=+3559.556990221" watchObservedRunningTime="2026-07-19 08:41:51.778435555 +0000 UTC m=+3559.558098869"